Training: 2021-03-15 09:30:45,143-rank_id: 0 Training: 2021-03-15 09:30:55,058-softmax weight init successfully! Training: 2021-03-15 09:30:55,058-softmax weight mom init successfully! Training: 2021-03-15 09:30:55,063-Total Step is: 124550 Training: 2021-03-15 09:31:34,976-Reducer buckets have been rebuilt in this iteration. Training: 2021-03-15 09:32:07,910-Speed 3209.48 samples/sec Loss 53.4353 Epoch: 0 Global Step: 100 Fp16 Grad Scale: 256 Required: 12 hours Training: 2021-03-15 09:32:24,541-Speed 3078.83 samples/sec Loss 52.3883 Epoch: 0 Global Step: 150 Fp16 Grad Scale: 256 Required: 12 hours Training: 2021-03-15 09:32:40,914-Speed 3127.18 samples/sec Loss 49.8980 Epoch: 0 Global Step: 200 Fp16 Grad Scale: 512 Required: 12 hours Training: 2021-03-15 09:32:57,120-Speed 3159.29 samples/sec Loss 47.7803 Epoch: 0 Global Step: 250 Fp16 Grad Scale: 512 Required: 12 hours Training: 2021-03-15 09:33:13,260-Speed 3172.49 samples/sec Loss 46.3488 Epoch: 0 Global Step: 300 Fp16 Grad Scale: 1024 Required: 12 hours Training: 2021-03-15 09:33:30,581-Speed 2956.06 samples/sec Loss 45.5286 Epoch: 0 Global Step: 350 Fp16 Grad Scale: 1024 Required: 12 hours Training: 2021-03-15 09:33:46,979-Speed 3122.46 samples/sec Loss 44.9390 Epoch: 0 Global Step: 400 Fp16 Grad Scale: 2048 Required: 12 hours Training: 2021-03-15 09:34:03,155-Speed 3165.30 samples/sec Loss 44.4824 Epoch: 0 Global Step: 450 Fp16 Grad Scale: 2048 Required: 12 hours Training: 2021-03-15 09:34:19,339-Speed 3163.56 samples/sec Loss 44.0042 Epoch: 0 Global Step: 500 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-03-15 09:34:35,554-Speed 3157.84 samples/sec Loss 43.5715 Epoch: 0 Global Step: 550 Fp16 Grad Scale: 4096 Required: 12 hours Training: 2021-03-15 09:34:51,780-Speed 3155.36 samples/sec Loss 43.0487 Epoch: 0 Global Step: 600 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-03-15 09:35:07,819-Speed 3192.39 samples/sec Loss 42.6295 Epoch: 0 Global Step: 650 Fp16 Grad Scale: 8192 Required: 11 hours Training: 2021-03-15 09:35:24,174-Speed 3130.61 samples/sec Loss 42.1733 Epoch: 0 Global Step: 700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:35:40,299-Speed 3175.26 samples/sec Loss 41.6618 Epoch: 0 Global Step: 750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:35:56,838-Speed 3095.80 samples/sec Loss 41.1394 Epoch: 0 Global Step: 800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:36:12,998-Speed 3168.56 samples/sec Loss 40.5991 Epoch: 0 Global Step: 850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:36:29,512-Speed 3100.48 samples/sec Loss 39.9688 Epoch: 0 Global Step: 900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:36:45,724-Speed 3158.13 samples/sec Loss 39.3651 Epoch: 0 Global Step: 950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:37:02,097-Speed 3127.25 samples/sec Loss 38.6727 Epoch: 0 Global Step: 1000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:37:18,280-Speed 3163.98 samples/sec Loss 37.9296 Epoch: 0 Global Step: 1050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:37:34,526-Speed 3151.67 samples/sec Loss 37.1406 Epoch: 0 Global Step: 1100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:37:50,699-Speed 3165.83 samples/sec Loss 36.3593 Epoch: 0 Global Step: 1150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:38:08,530-Speed 2871.43 samples/sec Loss 35.5019 Epoch: 0 Global Step: 1200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:38:25,003-Speed 3108.28 samples/sec Loss 34.4433 Epoch: 0 Global Step: 1250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:38:41,223-Speed 3156.68 samples/sec Loss 33.6308 Epoch: 0 Global Step: 1300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:38:57,503-Speed 3144.93 samples/sec Loss 32.7850 Epoch: 0 Global Step: 1350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:39:13,694-Speed 3162.34 samples/sec Loss 31.7492 Epoch: 0 Global Step: 1400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:39:29,828-Speed 3173.62 samples/sec Loss 30.7895 Epoch: 0 Global Step: 1450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:39:46,059-Speed 3154.46 samples/sec Loss 29.9093 Epoch: 0 Global Step: 1500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:40:02,287-Speed 3155.25 samples/sec Loss 28.9887 Epoch: 0 Global Step: 1550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:40:18,483-Speed 3161.39 samples/sec Loss 28.0921 Epoch: 0 Global Step: 1600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:40:34,779-Speed 3141.98 samples/sec Loss 27.3087 Epoch: 0 Global Step: 1650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:40:50,852-Speed 3185.52 samples/sec Loss 26.5823 Epoch: 0 Global Step: 1700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:41:07,209-Speed 3130.30 samples/sec Loss 25.8659 Epoch: 0 Global Step: 1750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:41:23,419-Speed 3158.64 samples/sec Loss 25.2512 Epoch: 0 Global Step: 1800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:41:39,799-Speed 3125.74 samples/sec Loss 24.5551 Epoch: 0 Global Step: 1850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:41:56,434-Speed 3078.03 samples/sec Loss 24.0698 Epoch: 0 Global Step: 1900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:42:12,650-Speed 3157.32 samples/sec Loss 23.5163 Epoch: 0 Global Step: 1950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:42:28,937-Speed 3143.77 samples/sec Loss 22.9513 Epoch: 0 Global Step: 2000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 09:43:24,803-[lfw][2000]XNorm: 21.965706 Training: 2021-03-15 09:43:24,804-[lfw][2000]Accuracy-Flip: 0.98067+-0.00692 Training: 2021-03-15 09:43:24,804-[lfw][2000]Accuracy-Highest: 0.98067 Training: 2021-03-15 09:44:29,664-[cfp_fp][2000]XNorm: 18.205391 Training: 2021-03-15 09:44:29,664-[cfp_fp][2000]Accuracy-Flip: 0.88186+-0.01618 Training: 2021-03-15 09:44:29,664-[cfp_fp][2000]Accuracy-Highest: 0.88186 Training: 2021-03-15 09:45:23,249-[agedb_30][2000]XNorm: 20.873791 Training: 2021-03-15 09:45:23,249-[agedb_30][2000]Accuracy-Flip: 0.88250+-0.02028 Training: 2021-03-15 09:45:23,249-[agedb_30][2000]Accuracy-Highest: 0.88250 Training: 2021-03-15 09:45:39,317-Speed 268.94 samples/sec Loss 22.5113 Epoch: 0 Global Step: 2050 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:45:55,422-Speed 3179.33 samples/sec Loss 21.9867 Epoch: 0 Global Step: 2100 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:46:11,673-Speed 3150.55 samples/sec Loss 21.8040 Epoch: 0 Global Step: 2150 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:46:28,399-Speed 3061.23 samples/sec Loss 21.3361 Epoch: 0 Global Step: 2200 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:46:44,870-Speed 3108.59 samples/sec Loss 20.9816 Epoch: 0 Global Step: 2250 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:47:01,352-Speed 3106.45 samples/sec Loss 20.6612 Epoch: 0 Global Step: 2300 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:47:18,275-Speed 3025.63 samples/sec Loss 20.3563 Epoch: 0 Global Step: 2350 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:47:34,954-Speed 3069.81 samples/sec Loss 20.1575 Epoch: 0 Global Step: 2400 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:47:51,996-Speed 3004.46 samples/sec Loss 19.8814 Epoch: 0 Global Step: 2450 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:48:08,149-Speed 3169.81 samples/sec Loss 19.6893 Epoch: 0 Global Step: 2500 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:48:24,697-Speed 3094.26 samples/sec Loss 19.4774 Epoch: 0 Global Step: 2550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:48:40,961-Speed 3148.01 samples/sec Loss 19.1284 Epoch: 0 Global Step: 2600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:48:57,210-Speed 3151.12 samples/sec Loss 18.9012 Epoch: 0 Global Step: 2650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:49:13,462-Speed 3150.55 samples/sec Loss 18.8190 Epoch: 0 Global Step: 2700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:49:29,815-Speed 3131.03 samples/sec Loss 18.5811 Epoch: 0 Global Step: 2750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:49:46,022-Speed 3159.22 samples/sec Loss 18.4878 Epoch: 0 Global Step: 2800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:50:02,383-Speed 3129.50 samples/sec Loss 18.4219 Epoch: 0 Global Step: 2850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:50:18,566-Speed 3163.86 samples/sec Loss 18.1728 Epoch: 0 Global Step: 2900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:50:35,162-Speed 3085.26 samples/sec Loss 17.9918 Epoch: 0 Global Step: 2950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:50:51,627-Speed 3109.63 samples/sec Loss 17.9227 Epoch: 0 Global Step: 3000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:51:07,910-Speed 3144.58 samples/sec Loss 17.8197 Epoch: 0 Global Step: 3050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:51:24,574-Speed 3072.59 samples/sec Loss 17.7736 Epoch: 0 Global Step: 3100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:51:41,458-Speed 3032.50 samples/sec Loss 17.4779 Epoch: 0 Global Step: 3150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:51:57,719-Speed 3148.81 samples/sec Loss 17.4782 Epoch: 0 Global Step: 3200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:52:14,037-Speed 3137.81 samples/sec Loss 17.3398 Epoch: 0 Global Step: 3250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:52:30,425-Speed 3124.21 samples/sec Loss 17.2110 Epoch: 0 Global Step: 3300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:52:46,855-Speed 3116.49 samples/sec Loss 17.1862 Epoch: 0 Global Step: 3350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:53:03,278-Speed 3117.52 samples/sec Loss 17.0360 Epoch: 0 Global Step: 3400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:53:19,625-Speed 3132.35 samples/sec Loss 16.9246 Epoch: 0 Global Step: 3450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:53:36,130-Speed 3102.20 samples/sec Loss 16.9041 Epoch: 0 Global Step: 3500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:53:52,513-Speed 3125.21 samples/sec Loss 16.8003 Epoch: 0 Global Step: 3550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:54:08,822-Speed 3139.49 samples/sec Loss 16.7322 Epoch: 0 Global Step: 3600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:54:24,951-Speed 3174.61 samples/sec Loss 16.5913 Epoch: 0 Global Step: 3650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:54:41,652-Speed 3065.67 samples/sec Loss 16.5222 Epoch: 0 Global Step: 3700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:54:58,585-Speed 3023.88 samples/sec Loss 16.5835 Epoch: 0 Global Step: 3750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:55:14,962-Speed 3126.40 samples/sec Loss 16.3839 Epoch: 0 Global Step: 3800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:55:31,256-Speed 3142.28 samples/sec Loss 16.3062 Epoch: 0 Global Step: 3850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:55:47,657-Speed 3122.03 samples/sec Loss 16.2247 Epoch: 0 Global Step: 3900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 09:56:03,892-Speed 3153.70 samples/sec Loss 16.1138 Epoch: 0 Global Step: 3950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 09:56:20,224-Speed 3135.14 samples/sec Loss 16.0437 Epoch: 0 Global Step: 4000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 09:57:13,311-[lfw][4000]XNorm: 21.015619 Training: 2021-03-15 09:57:13,311-[lfw][4000]Accuracy-Flip: 0.99300+-0.00332 Training: 2021-03-15 09:57:13,311-[lfw][4000]Accuracy-Highest: 0.99300 Training: 2021-03-15 09:58:15,059-[cfp_fp][4000]XNorm: 18.074456 Training: 2021-03-15 09:58:15,059-[cfp_fp][4000]Accuracy-Flip: 0.93529+-0.01409 Training: 2021-03-15 09:58:15,059-[cfp_fp][4000]Accuracy-Highest: 0.93529 Training: 2021-03-15 09:59:08,333-[agedb_30][4000]XNorm: 20.032625 Training: 2021-03-15 09:59:08,333-[agedb_30][4000]Accuracy-Flip: 0.93467+-0.01733 Training: 2021-03-15 09:59:08,333-[agedb_30][4000]Accuracy-Highest: 0.93467 Training: 2021-03-15 09:59:24,728-Speed 277.50 samples/sec Loss 16.0633 Epoch: 0 Global Step: 4050 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:59:40,938-Speed 3158.75 samples/sec Loss 16.0828 Epoch: 0 Global Step: 4100 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 09:59:57,134-Speed 3161.24 samples/sec Loss 16.0282 Epoch: 0 Global Step: 4150 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:00:13,287-Speed 3169.87 samples/sec Loss 15.9487 Epoch: 0 Global Step: 4200 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:00:29,798-Speed 3100.99 samples/sec Loss 16.0311 Epoch: 0 Global Step: 4250 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:00:45,871-Speed 3185.71 samples/sec Loss 15.8919 Epoch: 0 Global Step: 4300 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:01:02,366-Speed 3104.05 samples/sec Loss 15.7952 Epoch: 0 Global Step: 4350 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:01:18,487-Speed 3176.08 samples/sec Loss 15.7346 Epoch: 0 Global Step: 4400 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:01:34,782-Speed 3142.18 samples/sec Loss 15.7168 Epoch: 0 Global Step: 4450 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:01:51,654-Speed 3034.66 samples/sec Loss 15.7282 Epoch: 0 Global Step: 4500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:02:08,083-Speed 3116.60 samples/sec Loss 15.5424 Epoch: 0 Global Step: 4550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:02:24,395-Speed 3138.95 samples/sec Loss 15.5608 Epoch: 0 Global Step: 4600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:02:40,471-Speed 3184.96 samples/sec Loss 15.5097 Epoch: 0 Global Step: 4650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:02:56,734-Speed 3148.36 samples/sec Loss 15.5191 Epoch: 0 Global Step: 4700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:03:13,303-Speed 3090.26 samples/sec Loss 15.4593 Epoch: 0 Global Step: 4750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:03:29,478-Speed 3165.35 samples/sec Loss 15.4067 Epoch: 0 Global Step: 4800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:03:45,930-Speed 3112.31 samples/sec Loss 15.4201 Epoch: 0 Global Step: 4850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:04:02,165-Speed 3153.69 samples/sec Loss 15.3980 Epoch: 0 Global Step: 4900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:04:18,293-Speed 3174.75 samples/sec Loss 15.3200 Epoch: 0 Global Step: 4950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:04:37,161-Speed 2713.76 samples/sec Loss 15.0310 Epoch: 1 Global Step: 5000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:04:54,086-Speed 3025.24 samples/sec Loss 14.6322 Epoch: 1 Global Step: 5050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:05:10,687-Speed 3084.15 samples/sec Loss 14.6443 Epoch: 1 Global Step: 5100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:05:27,834-Speed 2986.06 samples/sec Loss 14.7001 Epoch: 1 Global Step: 5150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:05:44,152-Speed 3137.77 samples/sec Loss 14.6576 Epoch: 1 Global Step: 5200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:06:00,702-Speed 3093.92 samples/sec Loss 14.7953 Epoch: 1 Global Step: 5250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:06:16,942-Speed 3152.67 samples/sec Loss 14.8046 Epoch: 1 Global Step: 5300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:06:33,265-Speed 3136.94 samples/sec Loss 14.8476 Epoch: 1 Global Step: 5350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:06:49,550-Speed 3143.96 samples/sec Loss 14.9005 Epoch: 1 Global Step: 5400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:07:05,966-Speed 3119.16 samples/sec Loss 14.9412 Epoch: 1 Global Step: 5450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:07:22,380-Speed 3119.30 samples/sec Loss 14.8668 Epoch: 1 Global Step: 5500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:07:38,808-Speed 3116.88 samples/sec Loss 14.9258 Epoch: 1 Global Step: 5550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:07:54,953-Speed 3171.35 samples/sec Loss 14.9075 Epoch: 1 Global Step: 5600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:08:11,177-Speed 3155.93 samples/sec Loss 14.7444 Epoch: 1 Global Step: 5650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:08:27,506-Speed 3135.55 samples/sec Loss 14.8347 Epoch: 1 Global Step: 5700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:08:43,682-Speed 3165.33 samples/sec Loss 14.8344 Epoch: 1 Global Step: 5750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:09:00,036-Speed 3130.88 samples/sec Loss 14.7873 Epoch: 1 Global Step: 5800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:09:16,380-Speed 3132.70 samples/sec Loss 14.7178 Epoch: 1 Global Step: 5850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:09:33,311-Speed 3024.23 samples/sec Loss 14.7754 Epoch: 1 Global Step: 5900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:09:49,610-Speed 3141.38 samples/sec Loss 14.6640 Epoch: 1 Global Step: 5950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:10:05,820-Speed 3158.69 samples/sec Loss 14.7689 Epoch: 1 Global Step: 6000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:10:59,210-[lfw][6000]XNorm: 22.225221 Training: 2021-03-15 10:10:59,210-[lfw][6000]Accuracy-Flip: 0.99283+-0.00415 Training: 2021-03-15 10:10:59,211-[lfw][6000]Accuracy-Highest: 0.99300 Training: 2021-03-15 10:12:01,321-[cfp_fp][6000]XNorm: 19.738759 Training: 2021-03-15 10:12:01,321-[cfp_fp][6000]Accuracy-Flip: 0.94514+-0.01207 Training: 2021-03-15 10:12:01,321-[cfp_fp][6000]Accuracy-Highest: 0.94514 Training: 2021-03-15 10:12:54,806-[agedb_30][6000]XNorm: 21.742975 Training: 2021-03-15 10:12:54,806-[agedb_30][6000]Accuracy-Flip: 0.94233+-0.01177 Training: 2021-03-15 10:12:54,808-[agedb_30][6000]Accuracy-Highest: 0.94233 Training: 2021-03-15 10:13:11,141-Speed 276.28 samples/sec Loss 14.6792 Epoch: 1 Global Step: 6050 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:13:27,343-Speed 3160.15 samples/sec Loss 14.6717 Epoch: 1 Global Step: 6100 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:13:43,925-Speed 3087.91 samples/sec Loss 14.6793 Epoch: 1 Global Step: 6150 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:14:00,028-Speed 3179.61 samples/sec Loss 14.6752 Epoch: 1 Global Step: 6200 Fp16 Grad Scale: 16384 Required: 14 hours Training: 2021-03-15 10:14:16,379-Speed 3131.43 samples/sec Loss 14.5384 Epoch: 1 Global Step: 6250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:14:32,799-Speed 3118.34 samples/sec Loss 14.5666 Epoch: 1 Global Step: 6300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:14:49,232-Speed 3115.67 samples/sec Loss 14.5816 Epoch: 1 Global Step: 6350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:15:05,268-Speed 3192.89 samples/sec Loss 14.5391 Epoch: 1 Global Step: 6400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:15:21,498-Speed 3154.78 samples/sec Loss 14.5344 Epoch: 1 Global Step: 6450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:15:38,005-Speed 3101.97 samples/sec Loss 14.5322 Epoch: 1 Global Step: 6500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:15:54,362-Speed 3130.10 samples/sec Loss 14.5078 Epoch: 1 Global Step: 6550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:16:11,093-Speed 3060.37 samples/sec Loss 14.4663 Epoch: 1 Global Step: 6600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:16:27,566-Speed 3108.13 samples/sec Loss 14.4897 Epoch: 1 Global Step: 6650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:16:44,017-Speed 3112.45 samples/sec Loss 14.4487 Epoch: 1 Global Step: 6700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:17:00,457-Speed 3114.57 samples/sec Loss 14.4241 Epoch: 1 Global Step: 6750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:17:16,610-Speed 3169.75 samples/sec Loss 14.4195 Epoch: 1 Global Step: 6800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:17:33,035-Speed 3117.35 samples/sec Loss 14.2902 Epoch: 1 Global Step: 6850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:17:49,319-Speed 3144.22 samples/sec Loss 14.2582 Epoch: 1 Global Step: 6900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:18:05,734-Speed 3119.28 samples/sec Loss 14.2979 Epoch: 1 Global Step: 6950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:18:22,325-Speed 3086.10 samples/sec Loss 14.2642 Epoch: 1 Global Step: 7000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:18:38,781-Speed 3111.42 samples/sec Loss 14.3011 Epoch: 1 Global Step: 7050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:18:55,057-Speed 3145.90 samples/sec Loss 14.3134 Epoch: 1 Global Step: 7100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:19:11,513-Speed 3111.26 samples/sec Loss 14.1363 Epoch: 1 Global Step: 7150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:19:27,666-Speed 3169.82 samples/sec Loss 14.3114 Epoch: 1 Global Step: 7200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:19:44,033-Speed 3128.43 samples/sec Loss 14.2713 Epoch: 1 Global Step: 7250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:20:00,448-Speed 3119.25 samples/sec Loss 14.1333 Epoch: 1 Global Step: 7300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:20:17,078-Speed 3078.86 samples/sec Loss 14.2265 Epoch: 1 Global Step: 7350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:20:33,258-Speed 3164.52 samples/sec Loss 14.1512 Epoch: 1 Global Step: 7400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:20:49,604-Speed 3132.26 samples/sec Loss 14.1080 Epoch: 1 Global Step: 7450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:21:06,080-Speed 3107.75 samples/sec Loss 14.1329 Epoch: 1 Global Step: 7500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:21:22,500-Speed 3118.29 samples/sec Loss 14.2151 Epoch: 1 Global Step: 7550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:21:38,607-Speed 3178.70 samples/sec Loss 13.9908 Epoch: 1 Global Step: 7600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:21:54,767-Speed 3168.51 samples/sec Loss 13.9783 Epoch: 1 Global Step: 7650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:22:11,301-Speed 3096.69 samples/sec Loss 14.0650 Epoch: 1 Global Step: 7700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:22:27,450-Speed 3170.74 samples/sec Loss 14.0367 Epoch: 1 Global Step: 7750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:22:43,730-Speed 3144.98 samples/sec Loss 14.0826 Epoch: 1 Global Step: 7800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:23:00,028-Speed 3141.68 samples/sec Loss 14.0164 Epoch: 1 Global Step: 7850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:23:16,684-Speed 3073.99 samples/sec Loss 14.1255 Epoch: 1 Global Step: 7900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:23:32,891-Speed 3159.27 samples/sec Loss 14.0559 Epoch: 1 Global Step: 7950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:23:49,619-Speed 3060.95 samples/sec Loss 14.0453 Epoch: 1 Global Step: 8000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:24:42,951-[lfw][8000]XNorm: 22.814824 Training: 2021-03-15 10:24:42,952-[lfw][8000]Accuracy-Flip: 0.99417+-0.00344 Training: 2021-03-15 10:24:42,952-[lfw][8000]Accuracy-Highest: 0.99417 Training: 2021-03-15 10:25:44,690-[cfp_fp][8000]XNorm: 19.993162 Training: 2021-03-15 10:25:44,690-[cfp_fp][8000]Accuracy-Flip: 0.94571+-0.01178 Training: 2021-03-15 10:25:44,690-[cfp_fp][8000]Accuracy-Highest: 0.94571 Training: 2021-03-15 10:26:37,863-[agedb_30][8000]XNorm: 22.299625 Training: 2021-03-15 10:26:37,864-[agedb_30][8000]Accuracy-Flip: 0.95250+-0.00793 Training: 2021-03-15 10:26:37,864-[agedb_30][8000]Accuracy-Highest: 0.95250 Training: 2021-03-15 10:26:54,328-Speed 277.19 samples/sec Loss 13.9295 Epoch: 1 Global Step: 8050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:27:11,701-Speed 2947.20 samples/sec Loss 13.9970 Epoch: 1 Global Step: 8100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:27:28,058-Speed 3130.30 samples/sec Loss 13.9341 Epoch: 1 Global Step: 8150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:27:44,460-Speed 3121.55 samples/sec Loss 14.0275 Epoch: 1 Global Step: 8200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:28:00,626-Speed 3167.25 samples/sec Loss 13.9564 Epoch: 1 Global Step: 8250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:28:16,862-Speed 3153.72 samples/sec Loss 13.9020 Epoch: 1 Global Step: 8300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:28:33,345-Speed 3106.28 samples/sec Loss 13.9462 Epoch: 1 Global Step: 8350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:28:49,630-Speed 3144.13 samples/sec Loss 13.7831 Epoch: 1 Global Step: 8400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:29:06,103-Speed 3108.18 samples/sec Loss 13.9212 Epoch: 1 Global Step: 8450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:29:22,468-Speed 3128.68 samples/sec Loss 13.9074 Epoch: 1 Global Step: 8500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:29:38,738-Speed 3147.18 samples/sec Loss 13.8713 Epoch: 1 Global Step: 8550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:29:54,960-Speed 3156.15 samples/sec Loss 13.8479 Epoch: 1 Global Step: 8600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:30:11,274-Speed 3138.55 samples/sec Loss 13.8990 Epoch: 1 Global Step: 8650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:30:27,452-Speed 3165.01 samples/sec Loss 13.8370 Epoch: 1 Global Step: 8700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:30:43,522-Speed 3186.01 samples/sec Loss 13.8125 Epoch: 1 Global Step: 8750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:30:59,705-Speed 3164.06 samples/sec Loss 13.7389 Epoch: 1 Global Step: 8800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:31:16,372-Speed 3072.06 samples/sec Loss 13.7952 Epoch: 1 Global Step: 8850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:31:32,853-Speed 3106.61 samples/sec Loss 13.8252 Epoch: 1 Global Step: 8900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:31:49,179-Speed 3136.22 samples/sec Loss 13.8278 Epoch: 1 Global Step: 8950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:32:05,467-Speed 3143.62 samples/sec Loss 13.7113 Epoch: 1 Global Step: 9000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:32:22,068-Speed 3084.19 samples/sec Loss 13.7235 Epoch: 1 Global Step: 9050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:32:38,391-Speed 3136.85 samples/sec Loss 13.7361 Epoch: 1 Global Step: 9100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:32:54,605-Speed 3157.83 samples/sec Loss 13.8162 Epoch: 1 Global Step: 9150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:33:10,794-Speed 3162.68 samples/sec Loss 13.7750 Epoch: 1 Global Step: 9200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:33:27,242-Speed 3113.09 samples/sec Loss 13.6458 Epoch: 1 Global Step: 9250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:33:43,526-Speed 3144.31 samples/sec Loss 13.5839 Epoch: 1 Global Step: 9300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:33:59,929-Speed 3121.50 samples/sec Loss 13.6520 Epoch: 1 Global Step: 9350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:34:16,109-Speed 3164.42 samples/sec Loss 13.6883 Epoch: 1 Global Step: 9400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:34:32,298-Speed 3162.85 samples/sec Loss 13.6341 Epoch: 1 Global Step: 9450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:34:48,776-Speed 3107.30 samples/sec Loss 13.6207 Epoch: 1 Global Step: 9500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:35:05,615-Speed 3040.58 samples/sec Loss 13.7113 Epoch: 1 Global Step: 9550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:35:21,963-Speed 3132.06 samples/sec Loss 13.6116 Epoch: 1 Global Step: 9600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:35:39,282-Speed 2956.27 samples/sec Loss 13.5429 Epoch: 1 Global Step: 9650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:35:55,849-Speed 3090.72 samples/sec Loss 13.6205 Epoch: 1 Global Step: 9700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:36:12,148-Speed 3141.27 samples/sec Loss 13.5594 Epoch: 1 Global Step: 9750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:36:28,297-Speed 3170.75 samples/sec Loss 13.5755 Epoch: 1 Global Step: 9800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:36:44,464-Speed 3166.87 samples/sec Loss 13.5900 Epoch: 1 Global Step: 9850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:37:00,758-Speed 3142.49 samples/sec Loss 13.5766 Epoch: 1 Global Step: 9900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:37:17,405-Speed 3075.70 samples/sec Loss 13.5617 Epoch: 1 Global Step: 9950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:37:38,370-Speed 2442.25 samples/sec Loss 12.9754 Epoch: 2 Global Step: 10000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:38:31,836-[lfw][10000]XNorm: 23.017331 Training: 2021-03-15 10:38:31,836-[lfw][10000]Accuracy-Flip: 0.99500+-0.00373 Training: 2021-03-15 10:38:31,836-[lfw][10000]Accuracy-Highest: 0.99500 Training: 2021-03-15 10:39:33,659-[cfp_fp][10000]XNorm: 20.612630 Training: 2021-03-15 10:39:33,659-[cfp_fp][10000]Accuracy-Flip: 0.95500+-0.01347 Training: 2021-03-15 10:39:33,659-[cfp_fp][10000]Accuracy-Highest: 0.95500 Training: 2021-03-15 10:40:26,807-[agedb_30][10000]XNorm: 22.830887 Training: 2021-03-15 10:40:26,807-[agedb_30][10000]Accuracy-Flip: 0.95400+-0.01017 Training: 2021-03-15 10:40:26,807-[agedb_30][10000]Accuracy-Highest: 0.95400 Training: 2021-03-15 10:40:42,845-Speed 277.55 samples/sec Loss 12.8684 Epoch: 2 Global Step: 10050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:40:59,035-Speed 3162.58 samples/sec Loss 12.9899 Epoch: 2 Global Step: 10100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:41:15,229-Speed 3161.81 samples/sec Loss 13.0438 Epoch: 2 Global Step: 10150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:41:31,484-Speed 3149.80 samples/sec Loss 13.1622 Epoch: 2 Global Step: 10200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:41:48,161-Speed 3070.33 samples/sec Loss 13.3057 Epoch: 2 Global Step: 10250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:42:04,393-Speed 3154.37 samples/sec Loss 13.2985 Epoch: 2 Global Step: 10300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:42:20,619-Speed 3155.53 samples/sec Loss 13.3677 Epoch: 2 Global Step: 10350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:42:37,464-Speed 3039.62 samples/sec Loss 13.4415 Epoch: 2 Global Step: 10400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:42:53,708-Speed 3151.85 samples/sec Loss 13.4194 Epoch: 2 Global Step: 10450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:43:10,066-Speed 3130.11 samples/sec Loss 13.2972 Epoch: 2 Global Step: 10500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:43:26,456-Speed 3123.93 samples/sec Loss 13.4225 Epoch: 2 Global Step: 10550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:43:42,688-Speed 3154.52 samples/sec Loss 13.4733 Epoch: 2 Global Step: 10600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:43:58,991-Speed 3140.63 samples/sec Loss 13.3931 Epoch: 2 Global Step: 10650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:44:15,432-Speed 3114.22 samples/sec Loss 13.3849 Epoch: 2 Global Step: 10700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:44:31,723-Speed 3142.96 samples/sec Loss 13.4065 Epoch: 2 Global Step: 10750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:44:47,993-Speed 3146.89 samples/sec Loss 13.3702 Epoch: 2 Global Step: 10800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:45:04,259-Speed 3147.83 samples/sec Loss 13.4318 Epoch: 2 Global Step: 10850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:45:20,683-Speed 3117.53 samples/sec Loss 13.3919 Epoch: 2 Global Step: 10900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:45:37,091-Speed 3120.58 samples/sec Loss 13.3533 Epoch: 2 Global Step: 10950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:45:53,292-Speed 3160.33 samples/sec Loss 13.3054 Epoch: 2 Global Step: 11000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:46:09,702-Speed 3120.26 samples/sec Loss 13.3419 Epoch: 2 Global Step: 11050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:46:26,486-Speed 3050.55 samples/sec Loss 13.3587 Epoch: 2 Global Step: 11100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:46:43,149-Speed 3072.75 samples/sec Loss 13.3627 Epoch: 2 Global Step: 11150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:47:00,307-Speed 2984.15 samples/sec Loss 13.4024 Epoch: 2 Global Step: 11200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:47:16,673-Speed 3128.48 samples/sec Loss 13.2584 Epoch: 2 Global Step: 11250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:47:32,745-Speed 3185.78 samples/sec Loss 13.2780 Epoch: 2 Global Step: 11300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:47:48,888-Speed 3171.88 samples/sec Loss 13.3665 Epoch: 2 Global Step: 11350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:48:05,160-Speed 3146.64 samples/sec Loss 13.3519 Epoch: 2 Global Step: 11400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:48:21,373-Speed 3157.96 samples/sec Loss 13.3486 Epoch: 2 Global Step: 11450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:48:37,524-Speed 3170.25 samples/sec Loss 13.4640 Epoch: 2 Global Step: 11500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:48:54,010-Speed 3105.68 samples/sec Loss 13.3444 Epoch: 2 Global Step: 11550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:49:10,171-Speed 3168.28 samples/sec Loss 13.3450 Epoch: 2 Global Step: 11600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:49:26,508-Speed 3134.14 samples/sec Loss 13.3468 Epoch: 2 Global Step: 11650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:49:42,965-Speed 3111.17 samples/sec Loss 13.2565 Epoch: 2 Global Step: 11700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:49:59,584-Speed 3081.04 samples/sec Loss 13.3781 Epoch: 2 Global Step: 11750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:50:15,925-Speed 3133.30 samples/sec Loss 13.2524 Epoch: 2 Global Step: 11800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:50:32,050-Speed 3175.19 samples/sec Loss 13.2235 Epoch: 2 Global Step: 11850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:50:48,382-Speed 3135.06 samples/sec Loss 13.1850 Epoch: 2 Global Step: 11900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:51:04,531-Speed 3170.75 samples/sec Loss 13.2649 Epoch: 2 Global Step: 11950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 10:51:21,052-Speed 3099.16 samples/sec Loss 13.2741 Epoch: 2 Global Step: 12000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 10:52:14,449-[lfw][12000]XNorm: 21.995154 Training: 2021-03-15 10:52:14,449-[lfw][12000]Accuracy-Flip: 0.99467+-0.00379 Training: 2021-03-15 10:52:14,450-[lfw][12000]Accuracy-Highest: 0.99500 Training: 2021-03-15 10:53:16,297-[cfp_fp][12000]XNorm: 19.290335 Training: 2021-03-15 10:53:16,298-[cfp_fp][12000]Accuracy-Flip: 0.95014+-0.01154 Training: 2021-03-15 10:53:16,298-[cfp_fp][12000]Accuracy-Highest: 0.95500 Training: 2021-03-15 10:54:09,438-[agedb_30][12000]XNorm: 21.525875 Training: 2021-03-15 10:54:09,438-[agedb_30][12000]Accuracy-Flip: 0.95650+-0.00965 Training: 2021-03-15 10:54:09,438-[agedb_30][12000]Accuracy-Highest: 0.95650 Training: 2021-03-15 10:54:25,634-Speed 277.38 samples/sec Loss 13.3320 Epoch: 2 Global Step: 12050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:54:42,127-Speed 3104.54 samples/sec Loss 13.3336 Epoch: 2 Global Step: 12100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:54:58,467-Speed 3133.47 samples/sec Loss 13.1562 Epoch: 2 Global Step: 12150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:55:14,585-Speed 3176.77 samples/sec Loss 13.1501 Epoch: 2 Global Step: 12200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:55:30,755-Speed 3166.40 samples/sec Loss 13.1951 Epoch: 2 Global Step: 12250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:55:47,009-Speed 3150.17 samples/sec Loss 13.1555 Epoch: 2 Global Step: 12300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:56:03,204-Speed 3161.54 samples/sec Loss 13.1572 Epoch: 2 Global Step: 12350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:56:19,453-Speed 3151.05 samples/sec Loss 13.1046 Epoch: 2 Global Step: 12400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:56:35,944-Speed 3104.78 samples/sec Loss 13.1980 Epoch: 2 Global Step: 12450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:56:52,184-Speed 3152.89 samples/sec Loss 13.2304 Epoch: 2 Global Step: 12500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:57:08,315-Speed 3174.02 samples/sec Loss 13.1433 Epoch: 2 Global Step: 12550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:57:24,818-Speed 3102.59 samples/sec Loss 13.1137 Epoch: 2 Global Step: 12600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:57:41,352-Speed 3096.85 samples/sec Loss 13.1816 Epoch: 2 Global Step: 12650 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:57:58,174-Speed 3043.76 samples/sec Loss 13.1522 Epoch: 2 Global Step: 12700 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:58:15,016-Speed 3040.11 samples/sec Loss 13.2248 Epoch: 2 Global Step: 12750 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:58:32,126-Speed 2992.40 samples/sec Loss 13.1065 Epoch: 2 Global Step: 12800 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:58:48,445-Speed 3137.50 samples/sec Loss 13.1175 Epoch: 2 Global Step: 12850 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:59:04,650-Speed 3159.67 samples/sec Loss 13.1163 Epoch: 2 Global Step: 12900 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:59:20,833-Speed 3164.03 samples/sec Loss 13.1754 Epoch: 2 Global Step: 12950 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:59:37,253-Speed 3118.18 samples/sec Loss 13.0967 Epoch: 2 Global Step: 13000 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 10:59:53,642-Speed 3124.09 samples/sec Loss 13.0725 Epoch: 2 Global Step: 13050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:00:09,853-Speed 3158.57 samples/sec Loss 13.1454 Epoch: 2 Global Step: 13100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:00:26,360-Speed 3101.74 samples/sec Loss 13.1290 Epoch: 2 Global Step: 13150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:00:42,572-Speed 3158.31 samples/sec Loss 13.0986 Epoch: 2 Global Step: 13200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:00:58,783-Speed 3158.42 samples/sec Loss 13.1147 Epoch: 2 Global Step: 13250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:01:15,001-Speed 3157.17 samples/sec Loss 13.0747 Epoch: 2 Global Step: 13300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:01:31,250-Speed 3151.10 samples/sec Loss 13.0793 Epoch: 2 Global Step: 13350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:01:47,625-Speed 3126.74 samples/sec Loss 13.1355 Epoch: 2 Global Step: 13400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:02:04,056-Speed 3116.27 samples/sec Loss 13.1554 Epoch: 2 Global Step: 13450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:02:20,362-Speed 3139.91 samples/sec Loss 13.1058 Epoch: 2 Global Step: 13500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:02:36,820-Speed 3111.02 samples/sec Loss 12.9796 Epoch: 2 Global Step: 13550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:02:53,204-Speed 3125.18 samples/sec Loss 13.0419 Epoch: 2 Global Step: 13600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:03:09,407-Speed 3160.10 samples/sec Loss 13.0479 Epoch: 2 Global Step: 13650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:03:25,567-Speed 3168.30 samples/sec Loss 13.0600 Epoch: 2 Global Step: 13700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:03:41,761-Speed 3161.78 samples/sec Loss 13.1442 Epoch: 2 Global Step: 13750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:03:58,103-Speed 3133.11 samples/sec Loss 12.9845 Epoch: 2 Global Step: 13800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:04:14,365-Speed 3148.66 samples/sec Loss 13.0930 Epoch: 2 Global Step: 13850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:04:30,714-Speed 3131.65 samples/sec Loss 13.0437 Epoch: 2 Global Step: 13900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:04:47,201-Speed 3105.61 samples/sec Loss 13.1121 Epoch: 2 Global Step: 13950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:05:03,378-Speed 3165.23 samples/sec Loss 12.9436 Epoch: 2 Global Step: 14000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:05:56,546-[lfw][14000]XNorm: 23.154833 Training: 2021-03-15 11:05:56,546-[lfw][14000]Accuracy-Flip: 0.99483+-0.00345 Training: 2021-03-15 11:05:56,548-[lfw][14000]Accuracy-Highest: 0.99500 Training: 2021-03-15 11:06:58,381-[cfp_fp][14000]XNorm: 19.430887 Training: 2021-03-15 11:06:58,382-[cfp_fp][14000]Accuracy-Flip: 0.94600+-0.01059 Training: 2021-03-15 11:06:58,382-[cfp_fp][14000]Accuracy-Highest: 0.95500 Training: 2021-03-15 11:07:51,541-[agedb_30][14000]XNorm: 22.400216 Training: 2021-03-15 11:07:51,541-[agedb_30][14000]Accuracy-Flip: 0.95333+-0.00940 Training: 2021-03-15 11:07:51,541-[agedb_30][14000]Accuracy-Highest: 0.95650 Training: 2021-03-15 11:08:07,597-Speed 277.93 samples/sec Loss 13.0302 Epoch: 2 Global Step: 14050 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:08:23,901-Speed 3140.29 samples/sec Loss 13.0865 Epoch: 2 Global Step: 14100 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:08:40,070-Speed 3166.76 samples/sec Loss 13.0250 Epoch: 2 Global Step: 14150 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:08:56,829-Speed 3055.19 samples/sec Loss 13.1125 Epoch: 2 Global Step: 14200 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:09:13,302-Speed 3108.14 samples/sec Loss 13.0593 Epoch: 2 Global Step: 14250 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:09:30,351-Speed 3003.20 samples/sec Loss 12.9825 Epoch: 2 Global Step: 14300 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:09:46,779-Speed 3116.72 samples/sec Loss 13.0252 Epoch: 2 Global Step: 14350 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:10:03,271-Speed 3104.75 samples/sec Loss 12.9577 Epoch: 2 Global Step: 14400 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:10:19,640-Speed 3128.01 samples/sec Loss 12.9190 Epoch: 2 Global Step: 14450 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:10:35,810-Speed 3166.47 samples/sec Loss 12.9166 Epoch: 2 Global Step: 14500 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:10:51,971-Speed 3168.11 samples/sec Loss 13.1762 Epoch: 2 Global Step: 14550 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:11:08,134-Speed 3167.84 samples/sec Loss 13.0128 Epoch: 2 Global Step: 14600 Fp16 Grad Scale: 16384 Required: 13 hours Training: 2021-03-15 11:11:24,331-Speed 3161.27 samples/sec Loss 13.0773 Epoch: 2 Global Step: 14650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:11:40,589-Speed 3149.33 samples/sec Loss 12.9138 Epoch: 2 Global Step: 14700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:11:56,921-Speed 3134.98 samples/sec Loss 12.9555 Epoch: 2 Global Step: 14750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:12:13,330-Speed 3120.41 samples/sec Loss 12.9476 Epoch: 2 Global Step: 14800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:12:29,630-Speed 3141.06 samples/sec Loss 12.9242 Epoch: 2 Global Step: 14850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:12:46,146-Speed 3100.20 samples/sec Loss 12.9615 Epoch: 2 Global Step: 14900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:13:06,718-Speed 2488.86 samples/sec Loss 12.9514 Epoch: 3 Global Step: 14950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:13:23,216-Speed 3103.55 samples/sec Loss 12.2487 Epoch: 3 Global Step: 15000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:13:39,647-Speed 3116.17 samples/sec Loss 12.3097 Epoch: 3 Global Step: 15050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:13:55,779-Speed 3173.92 samples/sec Loss 12.4196 Epoch: 3 Global Step: 15100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:14:11,850-Speed 3186.09 samples/sec Loss 12.5620 Epoch: 3 Global Step: 15150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:14:28,230-Speed 3125.90 samples/sec Loss 12.5631 Epoch: 3 Global Step: 15200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:14:44,685-Speed 3111.60 samples/sec Loss 12.6202 Epoch: 3 Global Step: 15250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:15:01,054-Speed 3128.02 samples/sec Loss 12.6099 Epoch: 3 Global Step: 15300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:15:17,630-Speed 3088.82 samples/sec Loss 12.6870 Epoch: 3 Global Step: 15350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:15:33,902-Speed 3146.69 samples/sec Loss 12.7133 Epoch: 3 Global Step: 15400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:15:50,618-Speed 3063.15 samples/sec Loss 12.8206 Epoch: 3 Global Step: 15450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:16:06,763-Speed 3171.37 samples/sec Loss 12.8215 Epoch: 3 Global Step: 15500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:16:22,987-Speed 3155.94 samples/sec Loss 12.8468 Epoch: 3 Global Step: 15550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:16:39,424-Speed 3115.05 samples/sec Loss 12.8799 Epoch: 3 Global Step: 15600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:16:55,604-Speed 3164.45 samples/sec Loss 12.8092 Epoch: 3 Global Step: 15650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:17:11,846-Speed 3152.47 samples/sec Loss 12.8667 Epoch: 3 Global Step: 15700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:17:28,461-Speed 3081.63 samples/sec Loss 12.9023 Epoch: 3 Global Step: 15750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:17:44,887-Speed 3117.15 samples/sec Loss 12.8611 Epoch: 3 Global Step: 15800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:18:01,448-Speed 3091.70 samples/sec Loss 12.9294 Epoch: 3 Global Step: 15850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:18:18,741-Speed 2960.71 samples/sec Loss 12.7972 Epoch: 3 Global Step: 15900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:18:35,163-Speed 3118.02 samples/sec Loss 12.7779 Epoch: 3 Global Step: 15950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:18:51,450-Speed 3143.71 samples/sec Loss 12.7958 Epoch: 3 Global Step: 16000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:19:44,721-[lfw][16000]XNorm: 21.223189 Training: 2021-03-15 11:19:44,721-[lfw][16000]Accuracy-Flip: 0.99583+-0.00310 Training: 2021-03-15 11:19:44,721-[lfw][16000]Accuracy-Highest: 0.99583 Training: 2021-03-15 11:20:46,601-[cfp_fp][16000]XNorm: 18.487596 Training: 2021-03-15 11:20:46,601-[cfp_fp][16000]Accuracy-Flip: 0.95157+-0.01186 Training: 2021-03-15 11:20:46,601-[cfp_fp][16000]Accuracy-Highest: 0.95500 Training: 2021-03-15 11:21:39,823-[agedb_30][16000]XNorm: 20.795001 Training: 2021-03-15 11:21:39,823-[agedb_30][16000]Accuracy-Flip: 0.96117+-0.00972 Training: 2021-03-15 11:21:39,823-[agedb_30][16000]Accuracy-Highest: 0.96117 Training: 2021-03-15 11:21:55,843-Speed 277.67 samples/sec Loss 12.8190 Epoch: 3 Global Step: 16050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:22:11,998-Speed 3169.50 samples/sec Loss 12.8096 Epoch: 3 Global Step: 16100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:22:28,102-Speed 3179.28 samples/sec Loss 12.7788 Epoch: 3 Global Step: 16150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:22:44,269-Speed 3167.23 samples/sec Loss 12.7698 Epoch: 3 Global Step: 16200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:23:00,780-Speed 3100.89 samples/sec Loss 12.8378 Epoch: 3 Global Step: 16250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:23:17,000-Speed 3156.84 samples/sec Loss 12.7580 Epoch: 3 Global Step: 16300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:23:33,359-Speed 3129.94 samples/sec Loss 12.8883 Epoch: 3 Global Step: 16350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:23:49,693-Speed 3134.59 samples/sec Loss 12.8064 Epoch: 3 Global Step: 16400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:24:05,989-Speed 3141.97 samples/sec Loss 12.8210 Epoch: 3 Global Step: 16450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:24:22,093-Speed 3179.44 samples/sec Loss 12.8665 Epoch: 3 Global Step: 16500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:24:38,511-Speed 3118.69 samples/sec Loss 12.8759 Epoch: 3 Global Step: 16550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:24:54,659-Speed 3170.79 samples/sec Loss 12.8222 Epoch: 3 Global Step: 16600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:25:10,995-Speed 3134.19 samples/sec Loss 12.8619 Epoch: 3 Global Step: 16650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:25:27,680-Speed 3068.71 samples/sec Loss 12.7603 Epoch: 3 Global Step: 16700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:25:43,988-Speed 3139.69 samples/sec Loss 12.8615 Epoch: 3 Global Step: 16750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:26:00,201-Speed 3158.22 samples/sec Loss 12.8439 Epoch: 3 Global Step: 16800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:26:16,457-Speed 3149.55 samples/sec Loss 12.7481 Epoch: 3 Global Step: 16850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:26:32,869-Speed 3119.92 samples/sec Loss 12.8213 Epoch: 3 Global Step: 16900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:26:49,276-Speed 3120.57 samples/sec Loss 12.7719 Epoch: 3 Global Step: 16950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:27:05,969-Speed 3067.36 samples/sec Loss 12.7874 Epoch: 3 Global Step: 17000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:27:22,151-Speed 3164.18 samples/sec Loss 12.7842 Epoch: 3 Global Step: 17050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:27:38,204-Speed 3189.42 samples/sec Loss 12.7734 Epoch: 3 Global Step: 17100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:27:54,554-Speed 3131.67 samples/sec Loss 12.7334 Epoch: 3 Global Step: 17150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:28:10,719-Speed 3167.42 samples/sec Loss 12.7466 Epoch: 3 Global Step: 17200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:28:26,902-Speed 3163.87 samples/sec Loss 12.7425 Epoch: 3 Global Step: 17250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:28:43,216-Speed 3138.59 samples/sec Loss 12.7079 Epoch: 3 Global Step: 17300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:28:59,922-Speed 3064.84 samples/sec Loss 12.7463 Epoch: 3 Global Step: 17350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:29:16,415-Speed 3104.44 samples/sec Loss 12.7895 Epoch: 3 Global Step: 17400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:29:33,053-Speed 3077.52 samples/sec Loss 12.6920 Epoch: 3 Global Step: 17450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:29:50,261-Speed 2975.33 samples/sec Loss 12.7541 Epoch: 3 Global Step: 17500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:30:06,778-Speed 3100.00 samples/sec Loss 12.7441 Epoch: 3 Global Step: 17550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:30:23,037-Speed 3149.22 samples/sec Loss 12.7864 Epoch: 3 Global Step: 17600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:30:39,437-Speed 3122.06 samples/sec Loss 12.7796 Epoch: 3 Global Step: 17650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:30:55,599-Speed 3168.02 samples/sec Loss 12.8176 Epoch: 3 Global Step: 17700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:31:11,896-Speed 3141.82 samples/sec Loss 12.7479 Epoch: 3 Global Step: 17750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:31:28,259-Speed 3129.14 samples/sec Loss 12.8558 Epoch: 3 Global Step: 17800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:31:44,643-Speed 3125.05 samples/sec Loss 12.6767 Epoch: 3 Global Step: 17850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:32:00,902-Speed 3149.05 samples/sec Loss 12.7058 Epoch: 3 Global Step: 17900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:32:17,273-Speed 3127.65 samples/sec Loss 12.7872 Epoch: 3 Global Step: 17950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:32:33,622-Speed 3131.76 samples/sec Loss 12.7530 Epoch: 3 Global Step: 18000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:33:26,869-[lfw][18000]XNorm: 20.102720 Training: 2021-03-15 11:33:26,869-[lfw][18000]Accuracy-Flip: 0.99533+-0.00379 Training: 2021-03-15 11:33:26,870-[lfw][18000]Accuracy-Highest: 0.99583 Training: 2021-03-15 11:34:28,887-[cfp_fp][18000]XNorm: 17.277393 Training: 2021-03-15 11:34:28,887-[cfp_fp][18000]Accuracy-Flip: 0.95029+-0.00939 Training: 2021-03-15 11:34:28,887-[cfp_fp][18000]Accuracy-Highest: 0.95500 Training: 2021-03-15 11:35:22,296-[agedb_30][18000]XNorm: 19.659169 Training: 2021-03-15 11:35:22,297-[agedb_30][18000]Accuracy-Flip: 0.96233+-0.00712 Training: 2021-03-15 11:35:22,297-[agedb_30][18000]Accuracy-Highest: 0.96233 Training: 2021-03-15 11:35:38,464-Speed 276.99 samples/sec Loss 12.7447 Epoch: 3 Global Step: 18050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:35:54,636-Speed 3166.08 samples/sec Loss 12.7185 Epoch: 3 Global Step: 18100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:36:10,867-Speed 3154.61 samples/sec Loss 12.6508 Epoch: 3 Global Step: 18150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:36:27,231-Speed 3128.83 samples/sec Loss 12.8105 Epoch: 3 Global Step: 18200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:36:43,895-Speed 3072.60 samples/sec Loss 12.7370 Epoch: 3 Global Step: 18250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:37:00,069-Speed 3165.59 samples/sec Loss 12.6830 Epoch: 3 Global Step: 18300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:37:16,553-Speed 3106.32 samples/sec Loss 12.8067 Epoch: 3 Global Step: 18350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:37:32,973-Speed 3118.18 samples/sec Loss 12.7225 Epoch: 3 Global Step: 18400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:37:49,189-Speed 3157.58 samples/sec Loss 12.7668 Epoch: 3 Global Step: 18450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:38:05,416-Speed 3155.31 samples/sec Loss 12.6413 Epoch: 3 Global Step: 18500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:38:21,601-Speed 3163.39 samples/sec Loss 12.7045 Epoch: 3 Global Step: 18550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:38:38,045-Speed 3113.71 samples/sec Loss 12.7061 Epoch: 3 Global Step: 18600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:38:54,474-Speed 3116.63 samples/sec Loss 12.7111 Epoch: 3 Global Step: 18650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:39:10,890-Speed 3119.09 samples/sec Loss 12.6515 Epoch: 3 Global Step: 18700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:39:27,387-Speed 3103.68 samples/sec Loss 12.6394 Epoch: 3 Global Step: 18750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:39:43,742-Speed 3130.49 samples/sec Loss 12.7524 Epoch: 3 Global Step: 18800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:40:00,198-Speed 3111.48 samples/sec Loss 12.6220 Epoch: 3 Global Step: 18850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:40:16,511-Speed 3138.77 samples/sec Loss 12.7196 Epoch: 3 Global Step: 18900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:40:33,200-Speed 3067.92 samples/sec Loss 12.7326 Epoch: 3 Global Step: 18950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:40:49,548-Speed 3132.02 samples/sec Loss 12.7438 Epoch: 3 Global Step: 19000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:41:06,307-Speed 3055.17 samples/sec Loss 12.6733 Epoch: 3 Global Step: 19050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:41:23,521-Speed 2974.37 samples/sec Loss 12.5989 Epoch: 3 Global Step: 19100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:41:40,448-Speed 3024.94 samples/sec Loss 12.6361 Epoch: 3 Global Step: 19150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:41:57,220-Speed 3052.79 samples/sec Loss 12.6759 Epoch: 3 Global Step: 19200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:42:13,562-Speed 3133.15 samples/sec Loss 12.6250 Epoch: 3 Global Step: 19250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:42:29,906-Speed 3132.78 samples/sec Loss 12.6595 Epoch: 3 Global Step: 19300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:42:46,428-Speed 3098.90 samples/sec Loss 12.7245 Epoch: 3 Global Step: 19350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:43:02,568-Speed 3172.35 samples/sec Loss 12.5888 Epoch: 3 Global Step: 19400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:43:18,804-Speed 3153.62 samples/sec Loss 12.6525 Epoch: 3 Global Step: 19450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:43:35,233-Speed 3116.64 samples/sec Loss 12.6211 Epoch: 3 Global Step: 19500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:43:51,506-Speed 3146.28 samples/sec Loss 12.7458 Epoch: 3 Global Step: 19550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:44:07,966-Speed 3110.72 samples/sec Loss 12.6710 Epoch: 3 Global Step: 19600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:44:24,325-Speed 3129.96 samples/sec Loss 12.6742 Epoch: 3 Global Step: 19650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:44:40,596-Speed 3146.74 samples/sec Loss 12.6131 Epoch: 3 Global Step: 19700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:44:57,049-Speed 3111.97 samples/sec Loss 12.5749 Epoch: 3 Global Step: 19750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:45:13,400-Speed 3131.50 samples/sec Loss 12.5862 Epoch: 3 Global Step: 19800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:45:29,646-Speed 3151.58 samples/sec Loss 12.5434 Epoch: 3 Global Step: 19850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:45:45,944-Speed 3141.62 samples/sec Loss 12.6277 Epoch: 3 Global Step: 19900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:46:06,587-Speed 2480.39 samples/sec Loss 12.3047 Epoch: 4 Global Step: 19950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:46:23,131-Speed 3094.84 samples/sec Loss 11.9920 Epoch: 4 Global Step: 20000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:47:16,450-[lfw][20000]XNorm: 22.368430 Training: 2021-03-15 11:47:16,451-[lfw][20000]Accuracy-Flip: 0.99417+-0.00352 Training: 2021-03-15 11:47:16,451-[lfw][20000]Accuracy-Highest: 0.99583 Training: 2021-03-15 11:48:18,188-[cfp_fp][20000]XNorm: 19.380199 Training: 2021-03-15 11:48:18,189-[cfp_fp][20000]Accuracy-Flip: 0.95714+-0.00975 Training: 2021-03-15 11:48:18,189-[cfp_fp][20000]Accuracy-Highest: 0.95714 Training: 2021-03-15 11:49:11,183-[agedb_30][20000]XNorm: 21.798808 Training: 2021-03-15 11:49:11,183-[agedb_30][20000]Accuracy-Flip: 0.95933+-0.00746 Training: 2021-03-15 11:49:11,183-[agedb_30][20000]Accuracy-Highest: 0.96233 Training: 2021-03-15 11:49:27,233-Speed 278.11 samples/sec Loss 12.0175 Epoch: 4 Global Step: 20050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:49:43,308-Speed 3185.20 samples/sec Loss 12.1320 Epoch: 4 Global Step: 20100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:49:59,678-Speed 3127.93 samples/sec Loss 12.2661 Epoch: 4 Global Step: 20150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:50:15,786-Speed 3178.66 samples/sec Loss 12.3595 Epoch: 4 Global Step: 20200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:50:31,950-Speed 3167.58 samples/sec Loss 12.3090 Epoch: 4 Global Step: 20250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:50:48,096-Speed 3171.26 samples/sec Loss 12.3668 Epoch: 4 Global Step: 20300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:51:04,382-Speed 3143.90 samples/sec Loss 12.4122 Epoch: 4 Global Step: 20350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:51:20,762-Speed 3125.77 samples/sec Loss 12.4506 Epoch: 4 Global Step: 20400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:51:37,138-Speed 3126.62 samples/sec Loss 12.5033 Epoch: 4 Global Step: 20450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:51:53,291-Speed 3169.82 samples/sec Loss 12.5103 Epoch: 4 Global Step: 20500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:52:09,592-Speed 3141.05 samples/sec Loss 12.5810 Epoch: 4 Global Step: 20550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:52:26,387-Speed 3048.62 samples/sec Loss 12.5627 Epoch: 4 Global Step: 20600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:52:42,809-Speed 3117.94 samples/sec Loss 12.6557 Epoch: 4 Global Step: 20650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:53:00,230-Speed 2939.01 samples/sec Loss 12.5023 Epoch: 4 Global Step: 20700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:53:16,853-Speed 3080.25 samples/sec Loss 12.5011 Epoch: 4 Global Step: 20750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:53:33,321-Speed 3109.16 samples/sec Loss 12.5029 Epoch: 4 Global Step: 20800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:53:49,429-Speed 3178.66 samples/sec Loss 12.6012 Epoch: 4 Global Step: 20850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:54:05,605-Speed 3165.38 samples/sec Loss 12.4470 Epoch: 4 Global Step: 20900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:54:21,783-Speed 3164.91 samples/sec Loss 12.5000 Epoch: 4 Global Step: 20950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:54:38,100-Speed 3137.94 samples/sec Loss 12.5574 Epoch: 4 Global Step: 21000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:54:54,337-Speed 3153.30 samples/sec Loss 12.5123 Epoch: 4 Global Step: 21050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:55:10,590-Speed 3150.42 samples/sec Loss 12.5476 Epoch: 4 Global Step: 21100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:55:26,858-Speed 3147.33 samples/sec Loss 12.5544 Epoch: 4 Global Step: 21150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:55:43,034-Speed 3165.35 samples/sec Loss 12.5862 Epoch: 4 Global Step: 21200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:55:59,325-Speed 3142.80 samples/sec Loss 12.5186 Epoch: 4 Global Step: 21250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:56:15,497-Speed 3166.20 samples/sec Loss 12.4581 Epoch: 4 Global Step: 21300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:56:31,846-Speed 3131.76 samples/sec Loss 12.5949 Epoch: 4 Global Step: 21350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:56:48,326-Speed 3106.92 samples/sec Loss 12.5297 Epoch: 4 Global Step: 21400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:57:04,471-Speed 3171.38 samples/sec Loss 12.5538 Epoch: 4 Global Step: 21450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:57:20,793-Speed 3136.83 samples/sec Loss 12.4602 Epoch: 4 Global Step: 21500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:57:36,956-Speed 3167.86 samples/sec Loss 12.5383 Epoch: 4 Global Step: 21550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:57:53,443-Speed 3105.61 samples/sec Loss 12.5725 Epoch: 4 Global Step: 21600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:58:09,661-Speed 3157.19 samples/sec Loss 12.5611 Epoch: 4 Global Step: 21650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:58:25,856-Speed 3161.51 samples/sec Loss 12.5377 Epoch: 4 Global Step: 21700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:58:42,144-Speed 3143.64 samples/sec Loss 12.5040 Epoch: 4 Global Step: 21750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:58:58,872-Speed 3060.78 samples/sec Loss 12.4589 Epoch: 4 Global Step: 21800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:59:15,181-Speed 3139.39 samples/sec Loss 12.4620 Epoch: 4 Global Step: 21850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:59:31,389-Speed 3159.08 samples/sec Loss 12.4734 Epoch: 4 Global Step: 21900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 11:59:47,757-Speed 3128.16 samples/sec Loss 12.5365 Epoch: 4 Global Step: 21950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:00:03,908-Speed 3170.17 samples/sec Loss 12.5882 Epoch: 4 Global Step: 22000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:00:57,109-[lfw][22000]XNorm: 21.919361 Training: 2021-03-15 12:00:57,110-[lfw][22000]Accuracy-Flip: 0.99533+-0.00414 Training: 2021-03-15 12:00:57,110-[lfw][22000]Accuracy-Highest: 0.99583 Training: 2021-03-15 12:01:59,159-[cfp_fp][22000]XNorm: 18.482339 Training: 2021-03-15 12:01:59,160-[cfp_fp][22000]Accuracy-Flip: 0.95557+-0.01074 Training: 2021-03-15 12:01:59,160-[cfp_fp][22000]Accuracy-Highest: 0.95714 Training: 2021-03-15 12:02:52,568-[agedb_30][22000]XNorm: 21.466469 Training: 2021-03-15 12:02:52,568-[agedb_30][22000]Accuracy-Flip: 0.96367+-0.00991 Training: 2021-03-15 12:02:52,568-[agedb_30][22000]Accuracy-Highest: 0.96367 Training: 2021-03-15 12:03:08,788-Speed 276.94 samples/sec Loss 12.5593 Epoch: 4 Global Step: 22050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:03:24,931-Speed 3171.82 samples/sec Loss 12.4799 Epoch: 4 Global Step: 22100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:03:41,066-Speed 3173.39 samples/sec Loss 12.4247 Epoch: 4 Global Step: 22150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:03:57,604-Speed 3095.98 samples/sec Loss 12.4258 Epoch: 4 Global Step: 22200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:04:13,947-Speed 3132.90 samples/sec Loss 12.4204 Epoch: 4 Global Step: 22250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:04:30,788-Speed 3040.42 samples/sec Loss 12.4771 Epoch: 4 Global Step: 22300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:04:47,626-Speed 3040.73 samples/sec Loss 12.4565 Epoch: 4 Global Step: 22350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:05:04,055-Speed 3116.54 samples/sec Loss 12.4999 Epoch: 4 Global Step: 22400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:05:20,335-Speed 3145.08 samples/sec Loss 12.5321 Epoch: 4 Global Step: 22450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:05:36,700-Speed 3128.75 samples/sec Loss 12.4865 Epoch: 4 Global Step: 22500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:05:52,818-Speed 3176.66 samples/sec Loss 12.5138 Epoch: 4 Global Step: 22550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:06:08,937-Speed 3176.59 samples/sec Loss 12.4728 Epoch: 4 Global Step: 22600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:06:25,387-Speed 3112.50 samples/sec Loss 12.4981 Epoch: 4 Global Step: 22650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:06:41,629-Speed 3152.42 samples/sec Loss 12.5536 Epoch: 4 Global Step: 22700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:06:57,886-Speed 3149.61 samples/sec Loss 12.5516 Epoch: 4 Global Step: 22750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:07:14,090-Speed 3159.69 samples/sec Loss 12.4483 Epoch: 4 Global Step: 22800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:07:30,342-Speed 3150.62 samples/sec Loss 12.4039 Epoch: 4 Global Step: 22850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:07:46,632-Speed 3143.15 samples/sec Loss 12.4084 Epoch: 4 Global Step: 22900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:08:02,898-Speed 3147.76 samples/sec Loss 12.4575 Epoch: 4 Global Step: 22950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:08:19,180-Speed 3144.67 samples/sec Loss 12.5360 Epoch: 4 Global Step: 23000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:08:35,799-Speed 3080.85 samples/sec Loss 12.5141 Epoch: 4 Global Step: 23050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:08:52,052-Speed 3150.26 samples/sec Loss 12.4139 Epoch: 4 Global Step: 23100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:09:08,224-Speed 3166.13 samples/sec Loss 12.5361 Epoch: 4 Global Step: 23150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:09:24,660-Speed 3115.30 samples/sec Loss 12.4464 Epoch: 4 Global Step: 23200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:09:41,001-Speed 3133.34 samples/sec Loss 12.3691 Epoch: 4 Global Step: 23250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:09:57,451-Speed 3112.57 samples/sec Loss 12.5545 Epoch: 4 Global Step: 23300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:10:13,845-Speed 3123.04 samples/sec Loss 12.4635 Epoch: 4 Global Step: 23350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:10:30,218-Speed 3127.35 samples/sec Loss 12.4559 Epoch: 4 Global Step: 23400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:10:46,735-Speed 3099.97 samples/sec Loss 12.4573 Epoch: 4 Global Step: 23450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:11:03,120-Speed 3124.79 samples/sec Loss 12.3729 Epoch: 4 Global Step: 23500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:11:19,486-Speed 3128.65 samples/sec Loss 12.4041 Epoch: 4 Global Step: 23550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:11:35,726-Speed 3152.66 samples/sec Loss 12.3606 Epoch: 4 Global Step: 23600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:11:51,956-Speed 3154.91 samples/sec Loss 12.4551 Epoch: 4 Global Step: 23650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:12:08,260-Speed 3140.36 samples/sec Loss 12.4612 Epoch: 4 Global Step: 23700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:12:24,442-Speed 3164.22 samples/sec Loss 12.3610 Epoch: 4 Global Step: 23750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:12:41,235-Speed 3048.85 samples/sec Loss 12.3923 Epoch: 4 Global Step: 23800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:12:58,056-Speed 3044.05 samples/sec Loss 12.4072 Epoch: 4 Global Step: 23850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:13:14,994-Speed 3022.88 samples/sec Loss 12.4155 Epoch: 4 Global Step: 23900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:13:31,582-Speed 3086.71 samples/sec Loss 12.4710 Epoch: 4 Global Step: 23950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:13:48,095-Speed 3100.71 samples/sec Loss 12.4329 Epoch: 4 Global Step: 24000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:14:41,517-[lfw][24000]XNorm: 22.155141 Training: 2021-03-15 12:14:41,517-[lfw][24000]Accuracy-Flip: 0.99650+-0.00302 Training: 2021-03-15 12:14:41,519-[lfw][24000]Accuracy-Highest: 0.99650 Training: 2021-03-15 12:15:43,425-[cfp_fp][24000]XNorm: 19.102631 Training: 2021-03-15 12:15:43,425-[cfp_fp][24000]Accuracy-Flip: 0.96186+-0.00729 Training: 2021-03-15 12:15:43,425-[cfp_fp][24000]Accuracy-Highest: 0.96186 Training: 2021-03-15 12:16:36,718-[agedb_30][24000]XNorm: 21.733577 Training: 2021-03-15 12:16:36,718-[agedb_30][24000]Accuracy-Flip: 0.96500+-0.01101 Training: 2021-03-15 12:16:36,718-[agedb_30][24000]Accuracy-Highest: 0.96500 Training: 2021-03-15 12:16:52,806-Speed 277.19 samples/sec Loss 12.4298 Epoch: 4 Global Step: 24050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:17:08,820-Speed 3197.18 samples/sec Loss 12.4632 Epoch: 4 Global Step: 24100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 12:17:25,051-Speed 3154.75 samples/sec Loss 12.3550 Epoch: 4 Global Step: 24150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:17:41,666-Speed 3081.56 samples/sec Loss 12.4510 Epoch: 4 Global Step: 24200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:17:57,934-Speed 3147.46 samples/sec Loss 12.4419 Epoch: 4 Global Step: 24250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:18:14,144-Speed 3158.54 samples/sec Loss 12.4453 Epoch: 4 Global Step: 24300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:18:30,579-Speed 3115.56 samples/sec Loss 12.4111 Epoch: 4 Global Step: 24350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:18:46,912-Speed 3134.82 samples/sec Loss 12.4360 Epoch: 4 Global Step: 24400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:19:03,043-Speed 3174.02 samples/sec Loss 12.3476 Epoch: 4 Global Step: 24450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:19:19,151-Speed 3178.63 samples/sec Loss 12.3317 Epoch: 4 Global Step: 24500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:19:35,697-Speed 3094.59 samples/sec Loss 12.3471 Epoch: 4 Global Step: 24550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:19:52,225-Speed 3097.91 samples/sec Loss 12.3559 Epoch: 4 Global Step: 24600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:20:08,560-Speed 3134.46 samples/sec Loss 12.3866 Epoch: 4 Global Step: 24650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:20:24,949-Speed 3124.20 samples/sec Loss 12.4132 Epoch: 4 Global Step: 24700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:20:41,349-Speed 3121.93 samples/sec Loss 12.4285 Epoch: 4 Global Step: 24750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:20:57,539-Speed 3162.67 samples/sec Loss 12.4084 Epoch: 4 Global Step: 24800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:21:13,775-Speed 3153.59 samples/sec Loss 12.3908 Epoch: 4 Global Step: 24850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:21:30,079-Speed 3140.39 samples/sec Loss 12.4358 Epoch: 4 Global Step: 24900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:21:50,827-Speed 2467.76 samples/sec Loss 11.7949 Epoch: 5 Global Step: 24950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:22:07,070-Speed 3152.31 samples/sec Loss 11.6974 Epoch: 5 Global Step: 25000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:22:23,357-Speed 3143.59 samples/sec Loss 11.8710 Epoch: 5 Global Step: 25050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:22:39,772-Speed 3119.30 samples/sec Loss 11.8995 Epoch: 5 Global Step: 25100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:22:56,003-Speed 3154.67 samples/sec Loss 12.0490 Epoch: 5 Global Step: 25150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:23:12,520-Speed 3099.85 samples/sec Loss 12.1265 Epoch: 5 Global Step: 25200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:23:28,748-Speed 3155.11 samples/sec Loss 12.1236 Epoch: 5 Global Step: 25250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:23:45,009-Speed 3148.80 samples/sec Loss 12.2444 Epoch: 5 Global Step: 25300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:24:01,297-Speed 3143.54 samples/sec Loss 12.2604 Epoch: 5 Global Step: 25350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:24:17,546-Speed 3151.14 samples/sec Loss 12.3277 Epoch: 5 Global Step: 25400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:24:34,659-Speed 2991.82 samples/sec Loss 12.3105 Epoch: 5 Global Step: 25450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:24:51,087-Speed 3116.80 samples/sec Loss 12.3114 Epoch: 5 Global Step: 25500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:25:07,952-Speed 3035.90 samples/sec Loss 12.3573 Epoch: 5 Global Step: 25550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:25:24,569-Speed 3081.44 samples/sec Loss 12.2785 Epoch: 5 Global Step: 25600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:25:41,033-Speed 3109.76 samples/sec Loss 12.2920 Epoch: 5 Global Step: 25650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:25:58,104-Speed 2999.37 samples/sec Loss 12.3467 Epoch: 5 Global Step: 25700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:26:16,731-Speed 2748.77 samples/sec Loss 12.3480 Epoch: 5 Global Step: 25750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:26:35,403-Speed 2742.18 samples/sec Loss 12.3692 Epoch: 5 Global Step: 25800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:26:54,222-Speed 2720.84 samples/sec Loss 12.3487 Epoch: 5 Global Step: 25850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:27:12,858-Speed 2747.42 samples/sec Loss 12.3006 Epoch: 5 Global Step: 25900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:27:31,688-Speed 2719.11 samples/sec Loss 12.3923 Epoch: 5 Global Step: 25950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:27:50,606-Speed 2706.55 samples/sec Loss 12.3739 Epoch: 5 Global Step: 26000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:28:48,838-[lfw][26000]XNorm: 20.139129 Training: 2021-03-15 12:28:48,838-[lfw][26000]Accuracy-Flip: 0.99600+-0.00260 Training: 2021-03-15 12:28:48,838-[lfw][26000]Accuracy-Highest: 0.99650 Training: 2021-03-15 12:29:59,445-[cfp_fp][26000]XNorm: 17.644142 Training: 2021-03-15 12:29:59,445-[cfp_fp][26000]Accuracy-Flip: 0.96429+-0.01161 Training: 2021-03-15 12:29:59,445-[cfp_fp][26000]Accuracy-Highest: 0.96429 Training: 2021-03-15 12:31:02,902-[agedb_30][26000]XNorm: 19.956388 Training: 2021-03-15 12:31:02,902-[agedb_30][26000]Accuracy-Flip: 0.96133+-0.00918 Training: 2021-03-15 12:31:02,902-[agedb_30][26000]Accuracy-Highest: 0.96500 Training: 2021-03-15 12:31:28,546-Speed 234.93 samples/sec Loss 12.3111 Epoch: 5 Global Step: 26050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:31:53,924-Speed 2017.57 samples/sec Loss 12.3356 Epoch: 5 Global Step: 26100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:32:19,827-Speed 1976.71 samples/sec Loss 12.3080 Epoch: 5 Global Step: 26150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:32:45,629-Speed 1984.33 samples/sec Loss 12.3128 Epoch: 5 Global Step: 26200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:33:11,250-Speed 1998.48 samples/sec Loss 12.3477 Epoch: 5 Global Step: 26250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:33:36,771-Speed 2006.21 samples/sec Loss 12.2836 Epoch: 5 Global Step: 26300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:34:02,287-Speed 2006.67 samples/sec Loss 12.3489 Epoch: 5 Global Step: 26350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:34:28,074-Speed 1985.53 samples/sec Loss 12.3204 Epoch: 5 Global Step: 26400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:34:53,898-Speed 1982.82 samples/sec Loss 12.2678 Epoch: 5 Global Step: 26450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:35:19,626-Speed 1990.23 samples/sec Loss 12.2996 Epoch: 5 Global Step: 26500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:35:44,890-Speed 2026.67 samples/sec Loss 12.3299 Epoch: 5 Global Step: 26550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:36:10,420-Speed 2005.50 samples/sec Loss 12.3061 Epoch: 5 Global Step: 26600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:36:35,772-Speed 2019.68 samples/sec Loss 12.3756 Epoch: 5 Global Step: 26650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:37:01,383-Speed 1999.20 samples/sec Loss 12.3925 Epoch: 5 Global Step: 26700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:37:27,031-Speed 1996.38 samples/sec Loss 12.2537 Epoch: 5 Global Step: 26750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:37:52,454-Speed 2014.00 samples/sec Loss 12.3446 Epoch: 5 Global Step: 26800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:38:18,144-Speed 1993.13 samples/sec Loss 12.2399 Epoch: 5 Global Step: 26850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:38:43,791-Speed 1996.41 samples/sec Loss 12.2778 Epoch: 5 Global Step: 26900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:39:09,579-Speed 1985.45 samples/sec Loss 12.2813 Epoch: 5 Global Step: 26950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:39:34,927-Speed 2019.99 samples/sec Loss 12.3465 Epoch: 5 Global Step: 27000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:40:01,239-Speed 1945.94 samples/sec Loss 12.3370 Epoch: 5 Global Step: 27050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:40:27,610-Speed 1941.56 samples/sec Loss 12.2922 Epoch: 5 Global Step: 27100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:40:54,062-Speed 1935.65 samples/sec Loss 12.3552 Epoch: 5 Global Step: 27150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:41:19,867-Speed 1984.18 samples/sec Loss 12.3263 Epoch: 5 Global Step: 27200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:41:45,385-Speed 2006.49 samples/sec Loss 12.2976 Epoch: 5 Global Step: 27250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:42:11,336-Speed 1973.01 samples/sec Loss 12.3160 Epoch: 5 Global Step: 27300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:42:37,194-Speed 1980.15 samples/sec Loss 12.2899 Epoch: 5 Global Step: 27350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:43:02,953-Speed 1987.68 samples/sec Loss 12.2167 Epoch: 5 Global Step: 27400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:43:28,450-Speed 2008.16 samples/sec Loss 12.3325 Epoch: 5 Global Step: 27450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:43:54,184-Speed 1989.69 samples/sec Loss 12.2718 Epoch: 5 Global Step: 27500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:44:19,810-Speed 1998.04 samples/sec Loss 12.3052 Epoch: 5 Global Step: 27550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:44:45,444-Speed 1997.39 samples/sec Loss 12.3025 Epoch: 5 Global Step: 27600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:45:11,229-Speed 1985.74 samples/sec Loss 12.2665 Epoch: 5 Global Step: 27650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:45:36,577-Speed 2019.95 samples/sec Loss 12.2984 Epoch: 5 Global Step: 27700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:46:02,387-Speed 1983.75 samples/sec Loss 12.4557 Epoch: 5 Global Step: 27750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:46:28,147-Speed 1987.65 samples/sec Loss 12.2837 Epoch: 5 Global Step: 27800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:46:54,043-Speed 1977.21 samples/sec Loss 12.3328 Epoch: 5 Global Step: 27850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:47:20,188-Speed 1958.37 samples/sec Loss 12.1769 Epoch: 5 Global Step: 27900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:47:45,906-Speed 1990.92 samples/sec Loss 12.2298 Epoch: 5 Global Step: 27950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:48:11,621-Speed 1991.14 samples/sec Loss 12.2489 Epoch: 5 Global Step: 28000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:49:15,108-[lfw][28000]XNorm: 22.866981 Training: 2021-03-15 12:49:15,108-[lfw][28000]Accuracy-Flip: 0.99517+-0.00353 Training: 2021-03-15 12:49:15,108-[lfw][28000]Accuracy-Highest: 0.99650 Training: 2021-03-15 12:50:29,897-[cfp_fp][28000]XNorm: 19.679502 Training: 2021-03-15 12:50:29,897-[cfp_fp][28000]Accuracy-Flip: 0.95314+-0.00739 Training: 2021-03-15 12:50:29,897-[cfp_fp][28000]Accuracy-Highest: 0.96429 Training: 2021-03-15 12:51:33,822-[agedb_30][28000]XNorm: 22.254167 Training: 2021-03-15 12:51:33,822-[agedb_30][28000]Accuracy-Flip: 0.96050+-0.00922 Training: 2021-03-15 12:51:33,822-[agedb_30][28000]Accuracy-Highest: 0.96500 Training: 2021-03-15 12:51:59,410-Speed 224.77 samples/sec Loss 12.2570 Epoch: 5 Global Step: 28050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:52:24,619-Speed 2031.10 samples/sec Loss 12.2878 Epoch: 5 Global Step: 28100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:52:50,501-Speed 1978.26 samples/sec Loss 12.2896 Epoch: 5 Global Step: 28150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:53:16,115-Speed 1998.96 samples/sec Loss 12.3160 Epoch: 5 Global Step: 28200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:53:41,683-Speed 2002.54 samples/sec Loss 12.2685 Epoch: 5 Global Step: 28250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:54:07,054-Speed 2018.14 samples/sec Loss 12.2698 Epoch: 5 Global Step: 28300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:54:32,633-Speed 2001.68 samples/sec Loss 12.1760 Epoch: 5 Global Step: 28350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:54:58,163-Speed 2005.57 samples/sec Loss 12.2677 Epoch: 5 Global Step: 28400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:55:23,720-Speed 2003.42 samples/sec Loss 12.2828 Epoch: 5 Global Step: 28450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:55:49,113-Speed 2016.40 samples/sec Loss 12.2153 Epoch: 5 Global Step: 28500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:56:14,601-Speed 2008.91 samples/sec Loss 12.3205 Epoch: 5 Global Step: 28550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:56:40,587-Speed 1970.37 samples/sec Loss 12.1670 Epoch: 5 Global Step: 28600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:57:06,769-Speed 1955.66 samples/sec Loss 12.2039 Epoch: 5 Global Step: 28650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:57:32,379-Speed 1999.24 samples/sec Loss 12.3142 Epoch: 5 Global Step: 28700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:57:58,294-Speed 1975.78 samples/sec Loss 12.3197 Epoch: 5 Global Step: 28750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:58:24,473-Speed 1955.82 samples/sec Loss 12.2585 Epoch: 5 Global Step: 28800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:58:50,799-Speed 1944.92 samples/sec Loss 12.3208 Epoch: 5 Global Step: 28850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:59:16,349-Speed 2003.97 samples/sec Loss 12.2666 Epoch: 5 Global Step: 28900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 12:59:41,657-Speed 2023.17 samples/sec Loss 12.1643 Epoch: 5 Global Step: 28950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:00:06,894-Speed 2028.82 samples/sec Loss 12.3419 Epoch: 5 Global Step: 29000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:00:32,105-Speed 2030.93 samples/sec Loss 12.2010 Epoch: 5 Global Step: 29050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:00:58,110-Speed 1968.89 samples/sec Loss 12.2303 Epoch: 5 Global Step: 29100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:01:23,681-Speed 2002.39 samples/sec Loss 12.2031 Epoch: 5 Global Step: 29150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:01:49,068-Speed 2016.81 samples/sec Loss 12.1989 Epoch: 5 Global Step: 29200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:02:15,094-Speed 1967.36 samples/sec Loss 12.2285 Epoch: 5 Global Step: 29250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:02:40,531-Speed 2012.90 samples/sec Loss 12.2052 Epoch: 5 Global Step: 29300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:03:05,598-Speed 2042.56 samples/sec Loss 12.2977 Epoch: 5 Global Step: 29350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:03:31,203-Speed 1999.66 samples/sec Loss 12.2276 Epoch: 5 Global Step: 29400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:03:57,148-Speed 1973.50 samples/sec Loss 12.3096 Epoch: 5 Global Step: 29450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:04:22,855-Speed 1991.68 samples/sec Loss 12.2165 Epoch: 5 Global Step: 29500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:04:48,496-Speed 1996.92 samples/sec Loss 12.2274 Epoch: 5 Global Step: 29550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:05:14,084-Speed 2000.96 samples/sec Loss 12.1593 Epoch: 5 Global Step: 29600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:05:39,792-Speed 1991.71 samples/sec Loss 12.2511 Epoch: 5 Global Step: 29650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:06:05,557-Speed 1987.22 samples/sec Loss 12.1575 Epoch: 5 Global Step: 29700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:06:31,371-Speed 1983.47 samples/sec Loss 12.2976 Epoch: 5 Global Step: 29750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:06:56,894-Speed 2006.12 samples/sec Loss 12.2755 Epoch: 5 Global Step: 29800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:07:22,733-Speed 1981.59 samples/sec Loss 12.1890 Epoch: 5 Global Step: 29850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:07:53,933-Speed 1641.16 samples/sec Loss 12.1041 Epoch: 6 Global Step: 29900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:08:19,491-Speed 2003.50 samples/sec Loss 11.4980 Epoch: 6 Global Step: 29950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:08:45,421-Speed 1974.73 samples/sec Loss 11.5752 Epoch: 6 Global Step: 30000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:09:49,517-[lfw][30000]XNorm: 21.669758 Training: 2021-03-15 13:09:49,517-[lfw][30000]Accuracy-Flip: 0.99683+-0.00189 Training: 2021-03-15 13:09:49,518-[lfw][30000]Accuracy-Highest: 0.99683 Training: 2021-03-15 13:11:04,996-[cfp_fp][30000]XNorm: 18.644180 Training: 2021-03-15 13:11:04,997-[cfp_fp][30000]Accuracy-Flip: 0.96157+-0.01051 Training: 2021-03-15 13:11:04,997-[cfp_fp][30000]Accuracy-Highest: 0.96429 Training: 2021-03-15 13:12:09,709-[agedb_30][30000]XNorm: 21.165778 Training: 2021-03-15 13:12:09,710-[agedb_30][30000]Accuracy-Flip: 0.96233+-0.00786 Training: 2021-03-15 13:12:09,710-[agedb_30][30000]Accuracy-Highest: 0.96500 Training: 2021-03-15 13:12:35,130-Speed 222.89 samples/sec Loss 11.7799 Epoch: 6 Global Step: 30050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:13:00,613-Speed 2009.28 samples/sec Loss 11.8640 Epoch: 6 Global Step: 30100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:13:26,096-Speed 2009.21 samples/sec Loss 11.9294 Epoch: 6 Global Step: 30150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:13:51,759-Speed 1995.16 samples/sec Loss 11.9795 Epoch: 6 Global Step: 30200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:14:17,680-Speed 1975.31 samples/sec Loss 12.0913 Epoch: 6 Global Step: 30250 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:14:44,096-Speed 1938.28 samples/sec Loss 12.0326 Epoch: 6 Global Step: 30300 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:15:09,920-Speed 1982.72 samples/sec Loss 12.0694 Epoch: 6 Global Step: 30350 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:15:35,316-Speed 2016.13 samples/sec Loss 12.2042 Epoch: 6 Global Step: 30400 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:16:02,161-Speed 1907.28 samples/sec Loss 12.1412 Epoch: 6 Global Step: 30450 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:16:27,757-Speed 2000.37 samples/sec Loss 12.1653 Epoch: 6 Global Step: 30500 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:16:53,230-Speed 2010.11 samples/sec Loss 12.1561 Epoch: 6 Global Step: 30550 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:17:18,729-Speed 2007.98 samples/sec Loss 12.1446 Epoch: 6 Global Step: 30600 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:17:44,179-Speed 2011.81 samples/sec Loss 12.1887 Epoch: 6 Global Step: 30650 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:18:09,781-Speed 1999.95 samples/sec Loss 12.1792 Epoch: 6 Global Step: 30700 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:18:35,568-Speed 1985.52 samples/sec Loss 12.2172 Epoch: 6 Global Step: 30750 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:19:01,142-Speed 2002.12 samples/sec Loss 12.1851 Epoch: 6 Global Step: 30800 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:19:26,383-Speed 2028.52 samples/sec Loss 12.2336 Epoch: 6 Global Step: 30850 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:19:51,865-Speed 2009.41 samples/sec Loss 12.1746 Epoch: 6 Global Step: 30900 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:20:17,383-Speed 2006.62 samples/sec Loss 12.1906 Epoch: 6 Global Step: 30950 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:20:43,161-Speed 1986.25 samples/sec Loss 12.1689 Epoch: 6 Global Step: 31000 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:21:08,407-Speed 2028.07 samples/sec Loss 12.1382 Epoch: 6 Global Step: 31050 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:21:33,965-Speed 2003.41 samples/sec Loss 12.1776 Epoch: 6 Global Step: 31100 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:21:59,349-Speed 2017.09 samples/sec Loss 12.1786 Epoch: 6 Global Step: 31150 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:22:16,398-Speed 3003.25 samples/sec Loss 12.1494 Epoch: 6 Global Step: 31200 Fp16 Grad Scale: 16384 Required: 12 hours Training: 2021-03-15 13:22:32,574-Speed 3165.22 samples/sec Loss 12.1990 Epoch: 6 Global Step: 31250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:22:48,878-Speed 3140.48 samples/sec Loss 12.2220 Epoch: 6 Global Step: 31300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:23:05,229-Speed 3131.44 samples/sec Loss 12.1855 Epoch: 6 Global Step: 31350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:23:21,478-Speed 3151.05 samples/sec Loss 12.1732 Epoch: 6 Global Step: 31400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:23:37,758-Speed 3145.08 samples/sec Loss 12.1454 Epoch: 6 Global Step: 31450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:23:54,092-Speed 3134.61 samples/sec Loss 12.1435 Epoch: 6 Global Step: 31500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:24:10,273-Speed 3164.32 samples/sec Loss 12.2746 Epoch: 6 Global Step: 31550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:24:26,702-Speed 3116.58 samples/sec Loss 12.1703 Epoch: 6 Global Step: 31600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:24:42,825-Speed 3175.63 samples/sec Loss 12.1196 Epoch: 6 Global Step: 31650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:24:59,047-Speed 3156.31 samples/sec Loss 12.0801 Epoch: 6 Global Step: 31700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:25:15,287-Speed 3152.85 samples/sec Loss 12.0363 Epoch: 6 Global Step: 31750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:25:31,569-Speed 3144.65 samples/sec Loss 12.2640 Epoch: 6 Global Step: 31800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:25:47,939-Speed 3127.83 samples/sec Loss 12.1630 Epoch: 6 Global Step: 31850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:26:04,795-Speed 3037.70 samples/sec Loss 12.1210 Epoch: 6 Global Step: 31900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:26:21,342-Speed 3094.16 samples/sec Loss 12.1307 Epoch: 6 Global Step: 31950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:26:37,571-Speed 3155.01 samples/sec Loss 12.2167 Epoch: 6 Global Step: 32000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:27:30,824-[lfw][32000]XNorm: 23.794328 Training: 2021-03-15 13:27:30,824-[lfw][32000]Accuracy-Flip: 0.99567+-0.00249 Training: 2021-03-15 13:27:30,824-[lfw][32000]Accuracy-Highest: 0.99683 Training: 2021-03-15 13:28:32,759-[cfp_fp][32000]XNorm: 20.605423 Training: 2021-03-15 13:28:32,760-[cfp_fp][32000]Accuracy-Flip: 0.95357+-0.01299 Training: 2021-03-15 13:28:32,760-[cfp_fp][32000]Accuracy-Highest: 0.96429 Training: 2021-03-15 13:29:25,785-[agedb_30][32000]XNorm: 23.534625 Training: 2021-03-15 13:29:25,785-[agedb_30][32000]Accuracy-Flip: 0.95683+-0.00808 Training: 2021-03-15 13:29:25,786-[agedb_30][32000]Accuracy-Highest: 0.96500 Training: 2021-03-15 13:29:42,947-Speed 276.20 samples/sec Loss 12.2755 Epoch: 6 Global Step: 32050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:29:59,401-Speed 3111.82 samples/sec Loss 12.1663 Epoch: 6 Global Step: 32100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:30:15,568-Speed 3166.89 samples/sec Loss 12.1769 Epoch: 6 Global Step: 32150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:30:31,792-Speed 3155.99 samples/sec Loss 12.1386 Epoch: 6 Global Step: 32200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:30:48,163-Speed 3127.56 samples/sec Loss 12.1854 Epoch: 6 Global Step: 32250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:31:04,453-Speed 3143.27 samples/sec Loss 12.1750 Epoch: 6 Global Step: 32300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:31:21,356-Speed 3029.18 samples/sec Loss 12.2391 Epoch: 6 Global Step: 32350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:31:39,806-Speed 2775.18 samples/sec Loss 12.1518 Epoch: 6 Global Step: 32400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:31:59,018-Speed 2665.02 samples/sec Loss 12.1939 Epoch: 6 Global Step: 32450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:32:22,775-Speed 2155.35 samples/sec Loss 12.1575 Epoch: 6 Global Step: 32500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:32:48,708-Speed 1974.44 samples/sec Loss 12.1956 Epoch: 6 Global Step: 32550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:33:14,694-Speed 1970.32 samples/sec Loss 12.0902 Epoch: 6 Global Step: 32600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:33:40,635-Speed 1973.76 samples/sec Loss 12.0713 Epoch: 6 Global Step: 32650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:34:06,276-Speed 1996.93 samples/sec Loss 12.1795 Epoch: 6 Global Step: 32700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:34:31,805-Speed 2005.57 samples/sec Loss 12.0916 Epoch: 6 Global Step: 32750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:34:57,750-Speed 1973.50 samples/sec Loss 12.2037 Epoch: 6 Global Step: 32800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:35:23,303-Speed 2003.77 samples/sec Loss 12.1708 Epoch: 6 Global Step: 32850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:35:49,055-Speed 1988.23 samples/sec Loss 12.2210 Epoch: 6 Global Step: 32900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:36:14,696-Speed 1996.91 samples/sec Loss 12.1384 Epoch: 6 Global Step: 32950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:36:40,334-Speed 1997.08 samples/sec Loss 12.1150 Epoch: 6 Global Step: 33000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:37:06,325-Speed 1969.99 samples/sec Loss 12.1140 Epoch: 6 Global Step: 33050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:37:32,346-Speed 1967.71 samples/sec Loss 12.1586 Epoch: 6 Global Step: 33100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:37:58,022-Speed 1994.15 samples/sec Loss 12.0913 Epoch: 6 Global Step: 33150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:38:23,747-Speed 1990.37 samples/sec Loss 12.1067 Epoch: 6 Global Step: 33200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:38:49,692-Speed 1973.51 samples/sec Loss 12.1674 Epoch: 6 Global Step: 33250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:39:15,446-Speed 1988.09 samples/sec Loss 12.1046 Epoch: 6 Global Step: 33300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:39:41,196-Speed 1988.41 samples/sec Loss 12.1687 Epoch: 6 Global Step: 33350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:40:06,805-Speed 1999.37 samples/sec Loss 12.0607 Epoch: 6 Global Step: 33400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:40:32,804-Speed 1969.46 samples/sec Loss 12.1337 Epoch: 6 Global Step: 33450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:40:58,479-Speed 1994.23 samples/sec Loss 12.1301 Epoch: 6 Global Step: 33500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:41:24,899-Speed 1937.99 samples/sec Loss 12.1099 Epoch: 6 Global Step: 33550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:41:50,947-Speed 1965.63 samples/sec Loss 12.1116 Epoch: 6 Global Step: 33600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:42:16,895-Speed 1973.24 samples/sec Loss 12.1352 Epoch: 6 Global Step: 33650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:42:43,173-Speed 1948.47 samples/sec Loss 12.2076 Epoch: 6 Global Step: 33700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:43:09,103-Speed 1974.63 samples/sec Loss 12.0439 Epoch: 6 Global Step: 33750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:43:34,645-Speed 2004.58 samples/sec Loss 12.1301 Epoch: 6 Global Step: 33800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:44:00,297-Speed 1995.99 samples/sec Loss 12.0750 Epoch: 6 Global Step: 33850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:44:26,163-Speed 1979.53 samples/sec Loss 12.1231 Epoch: 6 Global Step: 33900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:44:51,576-Speed 2014.81 samples/sec Loss 12.1509 Epoch: 6 Global Step: 33950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:45:17,409-Speed 1982.03 samples/sec Loss 12.1624 Epoch: 6 Global Step: 34000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:46:21,982-[lfw][34000]XNorm: 20.207756 Training: 2021-03-15 13:46:21,982-[lfw][34000]Accuracy-Flip: 0.99517+-0.00369 Training: 2021-03-15 13:46:21,982-[lfw][34000]Accuracy-Highest: 0.99683 Training: 2021-03-15 13:47:36,173-[cfp_fp][34000]XNorm: 17.197712 Training: 2021-03-15 13:47:36,173-[cfp_fp][34000]Accuracy-Flip: 0.94971+-0.00699 Training: 2021-03-15 13:47:36,173-[cfp_fp][34000]Accuracy-Highest: 0.96429 Training: 2021-03-15 13:48:41,637-[agedb_30][34000]XNorm: 19.898033 Training: 2021-03-15 13:48:41,638-[agedb_30][34000]Accuracy-Flip: 0.96300+-0.00682 Training: 2021-03-15 13:48:41,638-[agedb_30][34000]Accuracy-Highest: 0.96500 Training: 2021-03-15 13:49:07,169-Speed 222.84 samples/sec Loss 12.0911 Epoch: 6 Global Step: 34050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:49:33,352-Speed 1955.50 samples/sec Loss 12.1147 Epoch: 6 Global Step: 34100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:49:58,935-Speed 2001.41 samples/sec Loss 12.2091 Epoch: 6 Global Step: 34150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:50:24,784-Speed 1980.83 samples/sec Loss 12.1612 Epoch: 6 Global Step: 34200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:50:50,950-Speed 1956.79 samples/sec Loss 12.1016 Epoch: 6 Global Step: 34250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:51:16,377-Speed 2013.69 samples/sec Loss 12.1367 Epoch: 6 Global Step: 34300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:51:42,120-Speed 1988.91 samples/sec Loss 12.0625 Epoch: 6 Global Step: 34350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:52:07,750-Speed 1997.75 samples/sec Loss 12.0847 Epoch: 6 Global Step: 34400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:52:33,338-Speed 2001.02 samples/sec Loss 12.0371 Epoch: 6 Global Step: 34450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:52:59,205-Speed 1979.37 samples/sec Loss 12.0712 Epoch: 6 Global Step: 34500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:53:24,836-Speed 1997.81 samples/sec Loss 12.0809 Epoch: 6 Global Step: 34550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:53:50,858-Speed 1967.56 samples/sec Loss 12.1590 Epoch: 6 Global Step: 34600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:54:16,598-Speed 1989.23 samples/sec Loss 12.0399 Epoch: 6 Global Step: 34650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:54:42,032-Speed 2013.08 samples/sec Loss 12.0804 Epoch: 6 Global Step: 34700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:55:07,833-Speed 1984.58 samples/sec Loss 12.0821 Epoch: 6 Global Step: 34750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:55:33,686-Speed 1980.45 samples/sec Loss 12.1071 Epoch: 6 Global Step: 34800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:55:59,375-Speed 1993.18 samples/sec Loss 12.1064 Epoch: 6 Global Step: 34850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:56:29,911-Speed 1676.77 samples/sec Loss 11.7331 Epoch: 7 Global Step: 34900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:56:55,486-Speed 2001.97 samples/sec Loss 11.5118 Epoch: 7 Global Step: 34950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:57:21,119-Speed 1997.50 samples/sec Loss 11.5525 Epoch: 7 Global Step: 35000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:57:46,936-Speed 1983.30 samples/sec Loss 11.6928 Epoch: 7 Global Step: 35050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:58:12,435-Speed 2007.97 samples/sec Loss 11.7202 Epoch: 7 Global Step: 35100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:58:38,326-Speed 1977.71 samples/sec Loss 11.8417 Epoch: 7 Global Step: 35150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:59:04,355-Speed 1967.18 samples/sec Loss 11.8885 Epoch: 7 Global Step: 35200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:59:30,495-Speed 1958.83 samples/sec Loss 11.9400 Epoch: 7 Global Step: 35250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 13:59:57,502-Speed 1895.86 samples/sec Loss 11.9921 Epoch: 7 Global Step: 35300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:00:23,259-Speed 1987.89 samples/sec Loss 11.9536 Epoch: 7 Global Step: 35350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:00:48,814-Speed 2003.64 samples/sec Loss 11.9368 Epoch: 7 Global Step: 35400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:01:14,493-Speed 1993.88 samples/sec Loss 12.0184 Epoch: 7 Global Step: 35450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:01:40,386-Speed 1977.45 samples/sec Loss 11.9953 Epoch: 7 Global Step: 35500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:02:05,980-Speed 2000.55 samples/sec Loss 12.1013 Epoch: 7 Global Step: 35550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:02:31,373-Speed 2016.35 samples/sec Loss 12.0280 Epoch: 7 Global Step: 35600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:02:57,061-Speed 1993.23 samples/sec Loss 11.9296 Epoch: 7 Global Step: 35650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:03:22,655-Speed 2000.54 samples/sec Loss 12.0395 Epoch: 7 Global Step: 35700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:03:48,564-Speed 1976.22 samples/sec Loss 12.0827 Epoch: 7 Global Step: 35750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:04:14,125-Speed 2003.12 samples/sec Loss 12.0972 Epoch: 7 Global Step: 35800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:04:39,867-Speed 1989.05 samples/sec Loss 12.0961 Epoch: 7 Global Step: 35850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:05:04,944-Speed 2041.80 samples/sec Loss 12.1329 Epoch: 7 Global Step: 35900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:05:30,458-Speed 2006.93 samples/sec Loss 12.1421 Epoch: 7 Global Step: 35950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:05:56,383-Speed 1974.95 samples/sec Loss 12.1740 Epoch: 7 Global Step: 36000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:07:02,517-[lfw][36000]XNorm: 20.842598 Training: 2021-03-15 14:07:02,517-[lfw][36000]Accuracy-Flip: 0.99517+-0.00283 Training: 2021-03-15 14:07:02,517-[lfw][36000]Accuracy-Highest: 0.99683 Training: 2021-03-15 14:08:16,806-[cfp_fp][36000]XNorm: 17.482771 Training: 2021-03-15 14:08:16,806-[cfp_fp][36000]Accuracy-Flip: 0.95257+-0.01420 Training: 2021-03-15 14:08:16,806-[cfp_fp][36000]Accuracy-Highest: 0.96429 Training: 2021-03-15 14:09:20,016-[agedb_30][36000]XNorm: 20.242509 Training: 2021-03-15 14:09:20,016-[agedb_30][36000]Accuracy-Flip: 0.95983+-0.00783 Training: 2021-03-15 14:09:20,016-[agedb_30][36000]Accuracy-Highest: 0.96500 Training: 2021-03-15 14:09:45,705-Speed 223.27 samples/sec Loss 12.1300 Epoch: 7 Global Step: 36050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:10:11,398-Speed 1992.84 samples/sec Loss 12.1529 Epoch: 7 Global Step: 36100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:10:37,227-Speed 1982.33 samples/sec Loss 12.0847 Epoch: 7 Global Step: 36150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:11:03,069-Speed 1981.30 samples/sec Loss 11.9020 Epoch: 7 Global Step: 36200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:11:28,892-Speed 1982.80 samples/sec Loss 12.0371 Epoch: 7 Global Step: 36250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:11:54,684-Speed 1985.20 samples/sec Loss 12.0220 Epoch: 7 Global Step: 36300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:12:20,422-Speed 1989.31 samples/sec Loss 12.1129 Epoch: 7 Global Step: 36350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:12:45,896-Speed 2009.99 samples/sec Loss 12.0816 Epoch: 7 Global Step: 36400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:13:11,273-Speed 2017.63 samples/sec Loss 12.0134 Epoch: 7 Global Step: 36450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:13:36,801-Speed 2005.70 samples/sec Loss 12.0951 Epoch: 7 Global Step: 36500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:14:02,315-Speed 2006.86 samples/sec Loss 12.0887 Epoch: 7 Global Step: 36550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:14:28,217-Speed 1976.77 samples/sec Loss 12.0858 Epoch: 7 Global Step: 36600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:14:54,429-Speed 1953.38 samples/sec Loss 12.0542 Epoch: 7 Global Step: 36650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:15:20,099-Speed 1994.60 samples/sec Loss 12.1330 Epoch: 7 Global Step: 36700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:15:46,284-Speed 1955.39 samples/sec Loss 12.0016 Epoch: 7 Global Step: 36750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:16:12,119-Speed 1981.85 samples/sec Loss 12.0527 Epoch: 7 Global Step: 36800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:16:39,094-Speed 1898.14 samples/sec Loss 12.0643 Epoch: 7 Global Step: 36850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:17:04,857-Speed 1987.41 samples/sec Loss 12.0908 Epoch: 7 Global Step: 36900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:17:31,020-Speed 1957.03 samples/sec Loss 11.9499 Epoch: 7 Global Step: 36950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:17:56,738-Speed 1990.94 samples/sec Loss 12.0037 Epoch: 7 Global Step: 37000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:18:22,431-Speed 1992.81 samples/sec Loss 12.0031 Epoch: 7 Global Step: 37050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:18:48,162-Speed 1989.87 samples/sec Loss 12.0191 Epoch: 7 Global Step: 37100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:19:13,903-Speed 1989.07 samples/sec Loss 12.1242 Epoch: 7 Global Step: 37150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:19:39,476-Speed 2002.22 samples/sec Loss 12.1097 Epoch: 7 Global Step: 37200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:20:04,984-Speed 2007.23 samples/sec Loss 12.0466 Epoch: 7 Global Step: 37250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:20:30,648-Speed 1995.07 samples/sec Loss 12.0335 Epoch: 7 Global Step: 37300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:20:56,469-Speed 1982.95 samples/sec Loss 12.0159 Epoch: 7 Global Step: 37350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:21:22,292-Speed 1982.92 samples/sec Loss 11.9644 Epoch: 7 Global Step: 37400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:21:47,864-Speed 2002.24 samples/sec Loss 11.9840 Epoch: 7 Global Step: 37450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:22:13,245-Speed 2017.38 samples/sec Loss 12.0198 Epoch: 7 Global Step: 37500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:22:38,755-Speed 2007.12 samples/sec Loss 12.0534 Epoch: 7 Global Step: 37550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:23:04,343-Speed 2001.02 samples/sec Loss 12.0333 Epoch: 7 Global Step: 37600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:23:29,469-Speed 2037.72 samples/sec Loss 12.0174 Epoch: 7 Global Step: 37650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:23:55,103-Speed 1997.44 samples/sec Loss 11.9976 Epoch: 7 Global Step: 37700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:24:20,669-Speed 2002.92 samples/sec Loss 12.0132 Epoch: 7 Global Step: 37750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:24:46,263-Speed 2000.57 samples/sec Loss 12.0506 Epoch: 7 Global Step: 37800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:25:11,917-Speed 1996.00 samples/sec Loss 12.1018 Epoch: 7 Global Step: 37850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:25:37,346-Speed 2013.46 samples/sec Loss 12.0874 Epoch: 7 Global Step: 37900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:26:03,166-Speed 1983.03 samples/sec Loss 12.0261 Epoch: 7 Global Step: 37950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:26:28,870-Speed 1992.02 samples/sec Loss 12.0102 Epoch: 7 Global Step: 38000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:27:32,844-[lfw][38000]XNorm: 21.558038 Training: 2021-03-15 14:27:32,844-[lfw][38000]Accuracy-Flip: 0.99650+-0.00311 Training: 2021-03-15 14:27:32,844-[lfw][38000]Accuracy-Highest: 0.99683 Training: 2021-03-15 14:28:47,266-[cfp_fp][38000]XNorm: 18.320922 Training: 2021-03-15 14:28:47,266-[cfp_fp][38000]Accuracy-Flip: 0.95657+-0.00751 Training: 2021-03-15 14:28:47,267-[cfp_fp][38000]Accuracy-Highest: 0.96429 Training: 2021-03-15 14:29:51,856-[agedb_30][38000]XNorm: 21.026750 Training: 2021-03-15 14:29:51,856-[agedb_30][38000]Accuracy-Flip: 0.96383+-0.00587 Training: 2021-03-15 14:29:51,856-[agedb_30][38000]Accuracy-Highest: 0.96500 Training: 2021-03-15 14:30:17,683-Speed 223.76 samples/sec Loss 12.1316 Epoch: 7 Global Step: 38050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:30:43,232-Speed 2004.16 samples/sec Loss 12.0717 Epoch: 7 Global Step: 38100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:31:08,891-Speed 1995.46 samples/sec Loss 12.0253 Epoch: 7 Global Step: 38150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:31:34,486-Speed 2000.47 samples/sec Loss 11.9349 Epoch: 7 Global Step: 38200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:32:00,491-Speed 1968.88 samples/sec Loss 12.0200 Epoch: 7 Global Step: 38250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:32:25,875-Speed 2017.13 samples/sec Loss 11.9984 Epoch: 7 Global Step: 38300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:32:51,565-Speed 1993.14 samples/sec Loss 12.0487 Epoch: 7 Global Step: 38350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:33:17,697-Speed 1959.38 samples/sec Loss 12.0248 Epoch: 7 Global Step: 38400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:33:43,477-Speed 1986.10 samples/sec Loss 11.9451 Epoch: 7 Global Step: 38450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:34:10,048-Speed 1926.98 samples/sec Loss 11.9821 Epoch: 7 Global Step: 38500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:34:36,208-Speed 1957.21 samples/sec Loss 12.0794 Epoch: 7 Global Step: 38550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:35:01,844-Speed 1997.31 samples/sec Loss 12.0460 Epoch: 7 Global Step: 38600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:35:27,682-Speed 1981.59 samples/sec Loss 12.0299 Epoch: 7 Global Step: 38650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:35:53,331-Speed 1996.27 samples/sec Loss 11.9748 Epoch: 7 Global Step: 38700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:36:18,753-Speed 2014.07 samples/sec Loss 12.0886 Epoch: 7 Global Step: 38750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:36:44,343-Speed 2000.82 samples/sec Loss 12.0291 Epoch: 7 Global Step: 38800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:37:10,165-Speed 1982.87 samples/sec Loss 12.0415 Epoch: 7 Global Step: 38850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:37:35,893-Speed 1990.13 samples/sec Loss 12.0524 Epoch: 7 Global Step: 38900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:38:01,599-Speed 1991.82 samples/sec Loss 11.9871 Epoch: 7 Global Step: 38950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:38:27,433-Speed 1981.92 samples/sec Loss 11.9023 Epoch: 7 Global Step: 39000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:38:53,267-Speed 1981.98 samples/sec Loss 12.0097 Epoch: 7 Global Step: 39050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:39:19,108-Speed 1981.43 samples/sec Loss 12.0067 Epoch: 7 Global Step: 39100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:39:44,554-Speed 2012.11 samples/sec Loss 11.8818 Epoch: 7 Global Step: 39150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:40:10,134-Speed 2001.66 samples/sec Loss 12.0341 Epoch: 7 Global Step: 39200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:40:35,874-Speed 1989.23 samples/sec Loss 12.0498 Epoch: 7 Global Step: 39250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:41:01,518-Speed 1996.63 samples/sec Loss 12.0640 Epoch: 7 Global Step: 39300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:41:27,230-Speed 1991.33 samples/sec Loss 11.9488 Epoch: 7 Global Step: 39350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:41:53,011-Speed 1986.05 samples/sec Loss 12.0099 Epoch: 7 Global Step: 39400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:42:18,723-Speed 1991.37 samples/sec Loss 12.0634 Epoch: 7 Global Step: 39450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:42:44,247-Speed 2005.98 samples/sec Loss 11.9909 Epoch: 7 Global Step: 39500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:43:09,991-Speed 1988.89 samples/sec Loss 11.9829 Epoch: 7 Global Step: 39550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:43:35,721-Speed 1989.95 samples/sec Loss 12.0354 Epoch: 7 Global Step: 39600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:44:01,173-Speed 2011.73 samples/sec Loss 12.0819 Epoch: 7 Global Step: 39650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:44:26,642-Speed 2010.34 samples/sec Loss 12.0009 Epoch: 7 Global Step: 39700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:44:52,348-Speed 1991.81 samples/sec Loss 11.8986 Epoch: 7 Global Step: 39750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:45:18,000-Speed 1996.06 samples/sec Loss 11.9793 Epoch: 7 Global Step: 39800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:45:44,297-Speed 1947.06 samples/sec Loss 12.0110 Epoch: 7 Global Step: 39850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:46:14,800-Speed 1678.55 samples/sec Loss 11.4145 Epoch: 8 Global Step: 39900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:46:40,525-Speed 1990.37 samples/sec Loss 11.3892 Epoch: 8 Global Step: 39950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:47:05,951-Speed 2013.90 samples/sec Loss 11.5660 Epoch: 8 Global Step: 40000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:48:10,211-[lfw][40000]XNorm: 21.466663 Training: 2021-03-15 14:48:10,211-[lfw][40000]Accuracy-Flip: 0.99717+-0.00279 Training: 2021-03-15 14:48:10,211-[lfw][40000]Accuracy-Highest: 0.99717 Training: 2021-03-15 14:49:25,650-[cfp_fp][40000]XNorm: 18.169301 Training: 2021-03-15 14:49:25,650-[cfp_fp][40000]Accuracy-Flip: 0.96057+-0.00665 Training: 2021-03-15 14:49:25,650-[cfp_fp][40000]Accuracy-Highest: 0.96429 Training: 2021-03-15 14:50:29,452-[agedb_30][40000]XNorm: 20.863936 Training: 2021-03-15 14:50:29,453-[agedb_30][40000]Accuracy-Flip: 0.96167+-0.00894 Training: 2021-03-15 14:50:29,453-[agedb_30][40000]Accuracy-Highest: 0.96500 Training: 2021-03-15 14:50:55,424-Speed 223.12 samples/sec Loss 11.5577 Epoch: 8 Global Step: 40050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:51:21,118-Speed 1992.72 samples/sec Loss 11.7314 Epoch: 8 Global Step: 40100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:51:47,285-Speed 1956.73 samples/sec Loss 11.7448 Epoch: 8 Global Step: 40150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:52:13,183-Speed 1977.27 samples/sec Loss 11.8399 Epoch: 8 Global Step: 40200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:52:39,191-Speed 1968.68 samples/sec Loss 11.9049 Epoch: 8 Global Step: 40250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:53:04,691-Speed 2007.89 samples/sec Loss 11.8024 Epoch: 8 Global Step: 40300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:53:30,057-Speed 2018.54 samples/sec Loss 11.8010 Epoch: 8 Global Step: 40350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:53:55,712-Speed 1995.73 samples/sec Loss 11.9152 Epoch: 8 Global Step: 40400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:54:21,416-Speed 1992.01 samples/sec Loss 11.9429 Epoch: 8 Global Step: 40450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:54:46,848-Speed 2013.23 samples/sec Loss 12.0406 Epoch: 8 Global Step: 40500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:55:12,649-Speed 1984.50 samples/sec Loss 11.9892 Epoch: 8 Global Step: 40550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:55:38,829-Speed 1955.75 samples/sec Loss 11.9601 Epoch: 8 Global Step: 40600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:56:04,815-Speed 1970.39 samples/sec Loss 11.9698 Epoch: 8 Global Step: 40650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:56:30,362-Speed 2004.16 samples/sec Loss 11.9819 Epoch: 8 Global Step: 40700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:56:56,493-Speed 1959.47 samples/sec Loss 11.9797 Epoch: 8 Global Step: 40750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:57:22,325-Speed 1982.15 samples/sec Loss 12.0322 Epoch: 8 Global Step: 40800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:57:48,018-Speed 1992.81 samples/sec Loss 11.9843 Epoch: 8 Global Step: 40850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:58:13,496-Speed 2009.59 samples/sec Loss 12.0026 Epoch: 8 Global Step: 40900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:58:39,331-Speed 1981.86 samples/sec Loss 11.9736 Epoch: 8 Global Step: 40950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:59:04,851-Speed 2006.34 samples/sec Loss 11.9060 Epoch: 8 Global Step: 41000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:59:30,404-Speed 2003.73 samples/sec Loss 11.9937 Epoch: 8 Global Step: 41050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 14:59:56,349-Speed 1973.49 samples/sec Loss 12.0193 Epoch: 8 Global Step: 41100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:00:21,987-Speed 1997.12 samples/sec Loss 11.9548 Epoch: 8 Global Step: 41150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:00:47,980-Speed 1969.83 samples/sec Loss 11.9543 Epoch: 8 Global Step: 41200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:01:13,724-Speed 1988.89 samples/sec Loss 12.0124 Epoch: 8 Global Step: 41250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:01:39,222-Speed 2008.21 samples/sec Loss 12.0328 Epoch: 8 Global Step: 41300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:02:04,826-Speed 1999.78 samples/sec Loss 12.0534 Epoch: 8 Global Step: 41350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:02:30,419-Speed 2000.57 samples/sec Loss 12.0085 Epoch: 8 Global Step: 41400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:02:55,824-Speed 2015.42 samples/sec Loss 12.0564 Epoch: 8 Global Step: 41450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:03:22,016-Speed 1954.96 samples/sec Loss 11.9448 Epoch: 8 Global Step: 41500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:03:47,222-Speed 2031.28 samples/sec Loss 12.0178 Epoch: 8 Global Step: 41550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:04:12,709-Speed 2008.98 samples/sec Loss 11.9555 Epoch: 8 Global Step: 41600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:04:38,688-Speed 1970.86 samples/sec Loss 11.9412 Epoch: 8 Global Step: 41650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:05:04,488-Speed 1984.68 samples/sec Loss 11.9622 Epoch: 8 Global Step: 41700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:05:30,565-Speed 1963.49 samples/sec Loss 11.9747 Epoch: 8 Global Step: 41750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:05:56,847-Speed 1948.16 samples/sec Loss 11.9588 Epoch: 8 Global Step: 41800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:06:22,522-Speed 1994.17 samples/sec Loss 11.9915 Epoch: 8 Global Step: 41850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:06:48,309-Speed 1985.54 samples/sec Loss 11.9509 Epoch: 8 Global Step: 41900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:07:14,296-Speed 1970.30 samples/sec Loss 11.9626 Epoch: 8 Global Step: 41950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:07:39,874-Speed 2001.82 samples/sec Loss 11.9669 Epoch: 8 Global Step: 42000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:08:43,009-[lfw][42000]XNorm: 21.669049 Training: 2021-03-15 15:08:43,009-[lfw][42000]Accuracy-Flip: 0.99583+-0.00300 Training: 2021-03-15 15:08:43,009-[lfw][42000]Accuracy-Highest: 0.99717 Training: 2021-03-15 15:09:56,289-[cfp_fp][42000]XNorm: 18.664055 Training: 2021-03-15 15:09:56,290-[cfp_fp][42000]Accuracy-Flip: 0.95071+-0.01140 Training: 2021-03-15 15:09:56,290-[cfp_fp][42000]Accuracy-Highest: 0.96429 Training: 2021-03-15 15:11:00,681-[agedb_30][42000]XNorm: 21.321470 Training: 2021-03-15 15:11:00,681-[agedb_30][42000]Accuracy-Flip: 0.96367+-0.01227 Training: 2021-03-15 15:11:00,681-[agedb_30][42000]Accuracy-Highest: 0.96500 Training: 2021-03-15 15:11:26,275-Speed 226.15 samples/sec Loss 11.9343 Epoch: 8 Global Step: 42050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:11:51,613-Speed 2020.72 samples/sec Loss 11.9648 Epoch: 8 Global Step: 42100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:12:17,307-Speed 1992.93 samples/sec Loss 12.0178 Epoch: 8 Global Step: 42150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:12:43,417-Speed 1961.01 samples/sec Loss 11.9981 Epoch: 8 Global Step: 42200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:13:09,072-Speed 1995.71 samples/sec Loss 11.9468 Epoch: 8 Global Step: 42250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:13:35,034-Speed 1972.16 samples/sec Loss 12.0216 Epoch: 8 Global Step: 42300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:14:00,805-Speed 1986.80 samples/sec Loss 11.9623 Epoch: 8 Global Step: 42350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:14:26,725-Speed 1975.38 samples/sec Loss 11.9902 Epoch: 8 Global Step: 42400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:14:52,435-Speed 1991.73 samples/sec Loss 11.9319 Epoch: 8 Global Step: 42450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:15:17,971-Speed 2005.06 samples/sec Loss 11.9180 Epoch: 8 Global Step: 42500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:15:43,938-Speed 1971.81 samples/sec Loss 11.9395 Epoch: 8 Global Step: 42550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:16:09,725-Speed 1985.55 samples/sec Loss 11.8875 Epoch: 8 Global Step: 42600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:16:35,736-Speed 1968.44 samples/sec Loss 11.8677 Epoch: 8 Global Step: 42650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:17:01,333-Speed 2000.40 samples/sec Loss 11.9561 Epoch: 8 Global Step: 42700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:17:27,174-Speed 1981.40 samples/sec Loss 12.0830 Epoch: 8 Global Step: 42750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:17:52,589-Speed 2014.69 samples/sec Loss 11.9355 Epoch: 8 Global Step: 42800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:18:18,393-Speed 1984.28 samples/sec Loss 11.9722 Epoch: 8 Global Step: 42850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:18:44,404-Speed 1968.46 samples/sec Loss 12.0949 Epoch: 8 Global Step: 42900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:19:10,309-Speed 1976.55 samples/sec Loss 11.8903 Epoch: 8 Global Step: 42950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:19:35,733-Speed 2013.84 samples/sec Loss 11.8506 Epoch: 8 Global Step: 43000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:20:01,770-Speed 1966.52 samples/sec Loss 11.9308 Epoch: 8 Global Step: 43050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:20:27,510-Speed 1989.20 samples/sec Loss 11.8836 Epoch: 8 Global Step: 43100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:20:53,258-Speed 1988.59 samples/sec Loss 11.9376 Epoch: 8 Global Step: 43150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:21:19,100-Speed 1981.29 samples/sec Loss 11.8643 Epoch: 8 Global Step: 43200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:21:44,556-Speed 2011.40 samples/sec Loss 11.9417 Epoch: 8 Global Step: 43250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:22:10,767-Speed 1953.51 samples/sec Loss 11.9509 Epoch: 8 Global Step: 43300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:22:36,604-Speed 1981.72 samples/sec Loss 11.8651 Epoch: 8 Global Step: 43350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:23:02,612-Speed 1968.85 samples/sec Loss 11.9146 Epoch: 8 Global Step: 43400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:23:28,609-Speed 1969.67 samples/sec Loss 11.9524 Epoch: 8 Global Step: 43450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:23:54,351-Speed 1989.10 samples/sec Loss 11.8836 Epoch: 8 Global Step: 43500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:24:20,292-Speed 1973.94 samples/sec Loss 11.8837 Epoch: 8 Global Step: 43550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:24:46,222-Speed 1974.61 samples/sec Loss 11.8816 Epoch: 8 Global Step: 43600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:25:12,186-Speed 1972.01 samples/sec Loss 11.9296 Epoch: 8 Global Step: 43650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:25:38,101-Speed 1975.81 samples/sec Loss 11.9720 Epoch: 8 Global Step: 43700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:26:03,944-Speed 1981.32 samples/sec Loss 11.9611 Epoch: 8 Global Step: 43750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:26:29,588-Speed 1996.61 samples/sec Loss 11.9611 Epoch: 8 Global Step: 43800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:26:55,358-Speed 1986.85 samples/sec Loss 11.8852 Epoch: 8 Global Step: 43850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:27:21,276-Speed 1975.65 samples/sec Loss 11.9605 Epoch: 8 Global Step: 43900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:27:46,881-Speed 1999.70 samples/sec Loss 11.9643 Epoch: 8 Global Step: 43950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:28:12,613-Speed 1989.76 samples/sec Loss 11.9289 Epoch: 8 Global Step: 44000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:29:18,153-[lfw][44000]XNorm: 23.866746 Training: 2021-03-15 15:29:18,153-[lfw][44000]Accuracy-Flip: 0.99500+-0.00459 Training: 2021-03-15 15:29:18,153-[lfw][44000]Accuracy-Highest: 0.99717 Training: 2021-03-15 15:30:32,694-[cfp_fp][44000]XNorm: 20.737418 Training: 2021-03-15 15:30:32,694-[cfp_fp][44000]Accuracy-Flip: 0.95571+-0.01040 Training: 2021-03-15 15:30:32,694-[cfp_fp][44000]Accuracy-Highest: 0.96429 Training: 2021-03-15 15:31:34,808-[agedb_30][44000]XNorm: 23.281932 Training: 2021-03-15 15:31:34,809-[agedb_30][44000]Accuracy-Flip: 0.96300+-0.00690 Training: 2021-03-15 15:31:34,809-[agedb_30][44000]Accuracy-Highest: 0.96500 Training: 2021-03-15 15:32:00,931-Speed 224.25 samples/sec Loss 11.9479 Epoch: 8 Global Step: 44050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:32:26,425-Speed 2008.38 samples/sec Loss 11.9721 Epoch: 8 Global Step: 44100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:32:51,810-Speed 2017.00 samples/sec Loss 12.0303 Epoch: 8 Global Step: 44150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:33:17,375-Speed 2002.79 samples/sec Loss 11.9376 Epoch: 8 Global Step: 44200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:33:43,312-Speed 1974.10 samples/sec Loss 11.9604 Epoch: 8 Global Step: 44250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:34:09,180-Speed 1979.39 samples/sec Loss 11.8804 Epoch: 8 Global Step: 44300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:34:34,644-Speed 2010.74 samples/sec Loss 11.8402 Epoch: 8 Global Step: 44350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:35:00,671-Speed 1967.27 samples/sec Loss 11.9457 Epoch: 8 Global Step: 44400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:35:26,254-Speed 2001.43 samples/sec Loss 11.9483 Epoch: 8 Global Step: 44450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:35:51,892-Speed 1997.09 samples/sec Loss 11.9189 Epoch: 8 Global Step: 44500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:36:17,631-Speed 1989.32 samples/sec Loss 11.9912 Epoch: 8 Global Step: 44550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:36:43,254-Speed 1998.24 samples/sec Loss 11.9306 Epoch: 8 Global Step: 44600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:37:08,819-Speed 2002.80 samples/sec Loss 11.9510 Epoch: 8 Global Step: 44650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:37:34,739-Speed 1975.47 samples/sec Loss 11.9432 Epoch: 8 Global Step: 44700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:38:00,733-Speed 1969.80 samples/sec Loss 11.9344 Epoch: 8 Global Step: 44750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:38:26,292-Speed 2003.25 samples/sec Loss 11.8684 Epoch: 8 Global Step: 44800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:38:56,671-Speed 1685.40 samples/sec Loss 11.7967 Epoch: 9 Global Step: 44850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:39:22,421-Speed 1988.43 samples/sec Loss 11.2161 Epoch: 9 Global Step: 44900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:39:48,388-Speed 1971.77 samples/sec Loss 11.3166 Epoch: 9 Global Step: 44950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:40:14,733-Speed 1943.52 samples/sec Loss 11.4049 Epoch: 9 Global Step: 45000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:40:40,939-Speed 1953.84 samples/sec Loss 11.5960 Epoch: 9 Global Step: 45050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:41:07,164-Speed 1952.39 samples/sec Loss 11.5852 Epoch: 9 Global Step: 45100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:41:32,850-Speed 1993.38 samples/sec Loss 11.7011 Epoch: 9 Global Step: 45150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:41:58,670-Speed 1982.99 samples/sec Loss 11.8502 Epoch: 9 Global Step: 45200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:42:24,065-Speed 2016.20 samples/sec Loss 11.7987 Epoch: 9 Global Step: 45250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:42:49,799-Speed 1989.64 samples/sec Loss 11.7735 Epoch: 9 Global Step: 45300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:43:15,592-Speed 1985.09 samples/sec Loss 11.8554 Epoch: 9 Global Step: 45350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:43:41,508-Speed 1975.70 samples/sec Loss 11.9232 Epoch: 9 Global Step: 45400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:44:07,108-Speed 2000.11 samples/sec Loss 11.8726 Epoch: 9 Global Step: 45450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:44:32,992-Speed 1978.11 samples/sec Loss 11.8373 Epoch: 9 Global Step: 45500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:44:58,825-Speed 1982.00 samples/sec Loss 11.8668 Epoch: 9 Global Step: 45550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:45:24,385-Speed 2003.23 samples/sec Loss 11.9147 Epoch: 9 Global Step: 45600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:45:49,970-Speed 2001.24 samples/sec Loss 11.8691 Epoch: 9 Global Step: 45650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:46:15,778-Speed 1983.96 samples/sec Loss 11.9250 Epoch: 9 Global Step: 45700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:46:41,498-Speed 1990.70 samples/sec Loss 11.8599 Epoch: 9 Global Step: 45750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:47:07,226-Speed 1990.12 samples/sec Loss 11.9295 Epoch: 9 Global Step: 45800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:47:32,740-Speed 2006.83 samples/sec Loss 11.8877 Epoch: 9 Global Step: 45850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:47:58,349-Speed 1999.38 samples/sec Loss 11.9161 Epoch: 9 Global Step: 45900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:48:23,822-Speed 2010.00 samples/sec Loss 11.8752 Epoch: 9 Global Step: 45950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:48:49,705-Speed 1978.18 samples/sec Loss 11.8265 Epoch: 9 Global Step: 46000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:49:52,182-[lfw][46000]XNorm: 23.341715 Training: 2021-03-15 15:49:52,182-[lfw][46000]Accuracy-Flip: 0.99517+-0.00474 Training: 2021-03-15 15:49:52,182-[lfw][46000]Accuracy-Highest: 0.99717 Training: 2021-03-15 15:51:05,909-[cfp_fp][46000]XNorm: 19.760878 Training: 2021-03-15 15:51:05,909-[cfp_fp][46000]Accuracy-Flip: 0.95886+-0.01020 Training: 2021-03-15 15:51:05,909-[cfp_fp][46000]Accuracy-Highest: 0.96429 Training: 2021-03-15 15:52:08,172-[agedb_30][46000]XNorm: 22.926617 Training: 2021-03-15 15:52:08,172-[agedb_30][46000]Accuracy-Flip: 0.96283+-0.00654 Training: 2021-03-15 15:52:08,172-[agedb_30][46000]Accuracy-Highest: 0.96500 Training: 2021-03-15 15:52:33,568-Speed 228.71 samples/sec Loss 11.7609 Epoch: 9 Global Step: 46050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:52:58,821-Speed 2027.52 samples/sec Loss 11.9141 Epoch: 9 Global Step: 46100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:53:24,751-Speed 1974.67 samples/sec Loss 11.8370 Epoch: 9 Global Step: 46150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:53:50,557-Speed 1984.06 samples/sec Loss 11.9460 Epoch: 9 Global Step: 46200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:54:16,268-Speed 1991.40 samples/sec Loss 11.8962 Epoch: 9 Global Step: 46250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:54:41,984-Speed 1991.17 samples/sec Loss 11.8754 Epoch: 9 Global Step: 46300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:55:07,702-Speed 1990.93 samples/sec Loss 11.9048 Epoch: 9 Global Step: 46350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:55:33,407-Speed 1991.95 samples/sec Loss 11.9338 Epoch: 9 Global Step: 46400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:55:58,993-Speed 2001.12 samples/sec Loss 11.9152 Epoch: 9 Global Step: 46450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:56:24,537-Speed 2004.47 samples/sec Loss 11.8851 Epoch: 9 Global Step: 46500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:56:50,262-Speed 1990.34 samples/sec Loss 11.8929 Epoch: 9 Global Step: 46550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:57:15,839-Speed 2001.88 samples/sec Loss 11.9293 Epoch: 9 Global Step: 46600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:57:42,578-Speed 1914.81 samples/sec Loss 11.9568 Epoch: 9 Global Step: 46650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:58:08,455-Speed 1978.67 samples/sec Loss 11.8822 Epoch: 9 Global Step: 46700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:58:34,408-Speed 1972.86 samples/sec Loss 11.9931 Epoch: 9 Global Step: 46750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:59:00,288-Speed 1978.47 samples/sec Loss 11.9013 Epoch: 9 Global Step: 46800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:59:26,067-Speed 1986.21 samples/sec Loss 11.9400 Epoch: 9 Global Step: 46850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 15:59:51,672-Speed 1999.69 samples/sec Loss 11.8148 Epoch: 9 Global Step: 46900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:00:17,455-Speed 1985.83 samples/sec Loss 11.8877 Epoch: 9 Global Step: 46950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:00:43,341-Speed 1978.05 samples/sec Loss 11.8202 Epoch: 9 Global Step: 47000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:01:08,727-Speed 2016.89 samples/sec Loss 11.9589 Epoch: 9 Global Step: 47050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:01:34,592-Speed 1979.57 samples/sec Loss 11.9964 Epoch: 9 Global Step: 47100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:02:00,800-Speed 1953.71 samples/sec Loss 11.8968 Epoch: 9 Global Step: 47150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:02:26,578-Speed 1986.27 samples/sec Loss 11.8551 Epoch: 9 Global Step: 47200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:02:52,294-Speed 1991.08 samples/sec Loss 11.8474 Epoch: 9 Global Step: 47250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:03:18,187-Speed 1977.40 samples/sec Loss 11.8968 Epoch: 9 Global Step: 47300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:03:43,873-Speed 1993.38 samples/sec Loss 11.7829 Epoch: 9 Global Step: 47350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:04:09,477-Speed 1999.73 samples/sec Loss 11.9344 Epoch: 9 Global Step: 47400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:04:35,056-Speed 2001.70 samples/sec Loss 11.8023 Epoch: 9 Global Step: 47450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:05:00,238-Speed 2033.34 samples/sec Loss 11.9066 Epoch: 9 Global Step: 47500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:05:26,149-Speed 1976.04 samples/sec Loss 11.9498 Epoch: 9 Global Step: 47550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:05:51,814-Speed 1995.03 samples/sec Loss 11.8706 Epoch: 9 Global Step: 47600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:06:17,486-Speed 1994.45 samples/sec Loss 11.8903 Epoch: 9 Global Step: 47650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:06:43,325-Speed 1981.51 samples/sec Loss 11.9316 Epoch: 9 Global Step: 47700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:07:09,186-Speed 1979.93 samples/sec Loss 11.9494 Epoch: 9 Global Step: 47750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:07:34,695-Speed 2007.16 samples/sec Loss 11.7421 Epoch: 9 Global Step: 47800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:08:00,899-Speed 1953.99 samples/sec Loss 11.9500 Epoch: 9 Global Step: 47850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:08:26,407-Speed 2007.25 samples/sec Loss 11.8560 Epoch: 9 Global Step: 47900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:08:52,334-Speed 1974.82 samples/sec Loss 11.8041 Epoch: 9 Global Step: 47950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:09:18,307-Speed 1971.40 samples/sec Loss 11.9228 Epoch: 9 Global Step: 48000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:10:22,374-[lfw][48000]XNorm: 22.410216 Training: 2021-03-15 16:10:22,375-[lfw][48000]Accuracy-Flip: 0.99517+-0.00353 Training: 2021-03-15 16:10:22,375-[lfw][48000]Accuracy-Highest: 0.99717 Training: 2021-03-15 16:11:37,256-[cfp_fp][48000]XNorm: 19.044390 Training: 2021-03-15 16:11:37,256-[cfp_fp][48000]Accuracy-Flip: 0.95414+-0.00791 Training: 2021-03-15 16:11:37,256-[cfp_fp][48000]Accuracy-Highest: 0.96429 Training: 2021-03-15 16:12:39,709-[agedb_30][48000]XNorm: 21.861275 Training: 2021-03-15 16:12:39,709-[agedb_30][48000]Accuracy-Flip: 0.96067+-0.00824 Training: 2021-03-15 16:12:39,709-[agedb_30][48000]Accuracy-Highest: 0.96500 Training: 2021-03-15 16:13:05,299-Speed 225.56 samples/sec Loss 11.8073 Epoch: 9 Global Step: 48050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:13:30,905-Speed 1999.66 samples/sec Loss 11.9896 Epoch: 9 Global Step: 48100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:13:57,005-Speed 1961.71 samples/sec Loss 11.7739 Epoch: 9 Global Step: 48150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:14:22,977-Speed 1971.43 samples/sec Loss 11.8412 Epoch: 9 Global Step: 48200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:14:48,825-Speed 1980.87 samples/sec Loss 11.8955 Epoch: 9 Global Step: 48250 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:15:15,096-Speed 1949.02 samples/sec Loss 11.9797 Epoch: 9 Global Step: 48300 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:15:40,848-Speed 1988.23 samples/sec Loss 11.8649 Epoch: 9 Global Step: 48350 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:16:06,440-Speed 2000.74 samples/sec Loss 11.8584 Epoch: 9 Global Step: 48400 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:16:32,225-Speed 1985.81 samples/sec Loss 11.9668 Epoch: 9 Global Step: 48450 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:16:58,324-Speed 1961.79 samples/sec Loss 11.9402 Epoch: 9 Global Step: 48500 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:17:23,997-Speed 1994.39 samples/sec Loss 11.9506 Epoch: 9 Global Step: 48550 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:17:49,833-Speed 1981.81 samples/sec Loss 11.8457 Epoch: 9 Global Step: 48600 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:18:15,332-Speed 2008.16 samples/sec Loss 11.9401 Epoch: 9 Global Step: 48650 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:18:41,491-Speed 1957.42 samples/sec Loss 11.8598 Epoch: 9 Global Step: 48700 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:19:07,122-Speed 1997.71 samples/sec Loss 12.0429 Epoch: 9 Global Step: 48750 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:19:33,152-Speed 1967.01 samples/sec Loss 11.8349 Epoch: 9 Global Step: 48800 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:19:58,495-Speed 2020.36 samples/sec Loss 11.8289 Epoch: 9 Global Step: 48850 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:20:23,828-Speed 2021.18 samples/sec Loss 11.9306 Epoch: 9 Global Step: 48900 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:20:49,654-Speed 1982.56 samples/sec Loss 11.7569 Epoch: 9 Global Step: 48950 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:21:15,361-Speed 1991.84 samples/sec Loss 11.8770 Epoch: 9 Global Step: 49000 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:21:41,210-Speed 1980.78 samples/sec Loss 11.9113 Epoch: 9 Global Step: 49050 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:22:06,951-Speed 1989.11 samples/sec Loss 11.8278 Epoch: 9 Global Step: 49100 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:22:32,941-Speed 1970.15 samples/sec Loss 11.9440 Epoch: 9 Global Step: 49150 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:22:58,543-Speed 1999.93 samples/sec Loss 11.9234 Epoch: 9 Global Step: 49200 Fp16 Grad Scale: 16384 Required: 11 hours Training: 2021-03-15 16:23:24,418-Speed 1978.79 samples/sec Loss 11.8095 Epoch: 9 Global Step: 49250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:23:50,421-Speed 1969.07 samples/sec Loss 11.9120 Epoch: 9 Global Step: 49300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:24:15,765-Speed 2020.27 samples/sec Loss 11.9001 Epoch: 9 Global Step: 49350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:24:41,355-Speed 2000.95 samples/sec Loss 11.8238 Epoch: 9 Global Step: 49400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:25:07,375-Speed 1967.77 samples/sec Loss 11.7954 Epoch: 9 Global Step: 49450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:25:33,196-Speed 1982.94 samples/sec Loss 11.8440 Epoch: 9 Global Step: 49500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:25:58,827-Speed 1997.64 samples/sec Loss 11.8922 Epoch: 9 Global Step: 49550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:26:24,882-Speed 1965.12 samples/sec Loss 11.8457 Epoch: 9 Global Step: 49600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:26:50,645-Speed 1987.42 samples/sec Loss 11.9223 Epoch: 9 Global Step: 49650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:27:16,462-Speed 1983.23 samples/sec Loss 11.9384 Epoch: 9 Global Step: 49700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:27:41,990-Speed 2005.75 samples/sec Loss 11.7633 Epoch: 9 Global Step: 49750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:28:07,703-Speed 1991.28 samples/sec Loss 11.8794 Epoch: 9 Global Step: 49800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:28:38,634-Speed 1655.41 samples/sec Loss 10.8429 Epoch: 10 Global Step: 49850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:29:04,389-Speed 1988.03 samples/sec Loss 9.2988 Epoch: 10 Global Step: 49900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:29:30,801-Speed 1938.57 samples/sec Loss 8.8545 Epoch: 10 Global Step: 49950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:29:56,663-Speed 1979.77 samples/sec Loss 8.5427 Epoch: 10 Global Step: 50000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:30:59,525-[lfw][50000]XNorm: 23.129891 Training: 2021-03-15 16:30:59,525-[lfw][50000]Accuracy-Flip: 0.99667+-0.00298 Training: 2021-03-15 16:30:59,525-[lfw][50000]Accuracy-Highest: 0.99717 Training: 2021-03-15 16:32:12,747-[cfp_fp][50000]XNorm: 20.201477 Training: 2021-03-15 16:32:12,747-[cfp_fp][50000]Accuracy-Flip: 0.97800+-0.00703 Training: 2021-03-15 16:32:12,747-[cfp_fp][50000]Accuracy-Highest: 0.97800 Training: 2021-03-15 16:33:14,239-[agedb_30][50000]XNorm: 22.821549 Training: 2021-03-15 16:33:14,239-[agedb_30][50000]Accuracy-Flip: 0.97467+-0.00726 Training: 2021-03-15 16:33:14,239-[agedb_30][50000]Accuracy-Highest: 0.97467 Training: 2021-03-15 16:33:39,775-Speed 229.48 samples/sec Loss 8.2442 Epoch: 10 Global Step: 50050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:34:05,455-Speed 1993.87 samples/sec Loss 7.9673 Epoch: 10 Global Step: 50100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:34:31,123-Speed 1994.72 samples/sec Loss 7.7519 Epoch: 10 Global Step: 50150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:34:57,083-Speed 1972.40 samples/sec Loss 7.5143 Epoch: 10 Global Step: 50200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:35:22,668-Speed 2001.22 samples/sec Loss 7.3517 Epoch: 10 Global Step: 50250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:35:48,249-Speed 2001.58 samples/sec Loss 7.1310 Epoch: 10 Global Step: 50300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:36:13,950-Speed 1992.20 samples/sec Loss 7.0953 Epoch: 10 Global Step: 50350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:36:39,992-Speed 1966.11 samples/sec Loss 6.9205 Epoch: 10 Global Step: 50400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:37:05,791-Speed 1984.67 samples/sec Loss 6.8147 Epoch: 10 Global Step: 50450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:37:31,112-Speed 2022.08 samples/sec Loss 6.6810 Epoch: 10 Global Step: 50500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:37:56,531-Speed 2014.33 samples/sec Loss 6.6161 Epoch: 10 Global Step: 50550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:38:22,461-Speed 1974.66 samples/sec Loss 6.4821 Epoch: 10 Global Step: 50600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:38:47,663-Speed 2031.61 samples/sec Loss 6.4042 Epoch: 10 Global Step: 50650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:39:13,275-Speed 1999.12 samples/sec Loss 6.1622 Epoch: 10 Global Step: 50700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:39:39,204-Speed 1974.84 samples/sec Loss 6.0791 Epoch: 10 Global Step: 50750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:40:04,852-Speed 1996.41 samples/sec Loss 6.0060 Epoch: 10 Global Step: 50800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:40:30,751-Speed 1977.17 samples/sec Loss 5.9847 Epoch: 10 Global Step: 50850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:40:56,897-Speed 1958.30 samples/sec Loss 5.8648 Epoch: 10 Global Step: 50900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:41:22,801-Speed 1976.64 samples/sec Loss 5.7623 Epoch: 10 Global Step: 50950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:41:48,503-Speed 1992.11 samples/sec Loss 5.7184 Epoch: 10 Global Step: 51000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:42:14,554-Speed 1965.43 samples/sec Loss 5.6665 Epoch: 10 Global Step: 51050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:42:40,230-Speed 1994.11 samples/sec Loss 5.5939 Epoch: 10 Global Step: 51100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:43:05,990-Speed 1987.62 samples/sec Loss 5.4625 Epoch: 10 Global Step: 51150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:43:32,064-Speed 1963.73 samples/sec Loss 5.4676 Epoch: 10 Global Step: 51200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:43:57,686-Speed 1998.33 samples/sec Loss 5.3480 Epoch: 10 Global Step: 51250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:44:23,328-Speed 1996.81 samples/sec Loss 5.3995 Epoch: 10 Global Step: 51300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:44:48,909-Speed 2001.51 samples/sec Loss 5.2269 Epoch: 10 Global Step: 51350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:45:15,073-Speed 1956.96 samples/sec Loss 5.1968 Epoch: 10 Global Step: 51400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:45:40,955-Speed 1978.29 samples/sec Loss 5.2020 Epoch: 10 Global Step: 51450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:46:06,961-Speed 1968.82 samples/sec Loss 5.1150 Epoch: 10 Global Step: 51500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:46:32,974-Speed 1968.32 samples/sec Loss 5.0735 Epoch: 10 Global Step: 51550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:46:59,208-Speed 1951.68 samples/sec Loss 5.0258 Epoch: 10 Global Step: 51600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:47:25,186-Speed 1970.96 samples/sec Loss 4.9303 Epoch: 10 Global Step: 51650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:47:51,067-Speed 1978.34 samples/sec Loss 4.9253 Epoch: 10 Global Step: 51700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:48:16,574-Speed 2007.41 samples/sec Loss 4.9348 Epoch: 10 Global Step: 51750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:48:42,451-Speed 1978.61 samples/sec Loss 4.8282 Epoch: 10 Global Step: 51800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:49:08,232-Speed 1986.07 samples/sec Loss 4.8083 Epoch: 10 Global Step: 51850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:49:33,972-Speed 1989.17 samples/sec Loss 4.7341 Epoch: 10 Global Step: 51900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:49:59,754-Speed 1985.95 samples/sec Loss 4.7641 Epoch: 10 Global Step: 51950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:50:25,539-Speed 1985.69 samples/sec Loss 4.7169 Epoch: 10 Global Step: 52000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:51:29,599-[lfw][52000]XNorm: 22.652349 Training: 2021-03-15 16:51:29,600-[lfw][52000]Accuracy-Flip: 0.99717+-0.00269 Training: 2021-03-15 16:51:29,600-[lfw][52000]Accuracy-Highest: 0.99717 Training: 2021-03-15 16:52:43,168-[cfp_fp][52000]XNorm: 20.328642 Training: 2021-03-15 16:52:43,168-[cfp_fp][52000]Accuracy-Flip: 0.98657+-0.00249 Training: 2021-03-15 16:52:43,168-[cfp_fp][52000]Accuracy-Highest: 0.98657 Training: 2021-03-15 16:53:45,046-[agedb_30][52000]XNorm: 22.790737 Training: 2021-03-15 16:53:45,046-[agedb_30][52000]Accuracy-Flip: 0.97800+-0.00865 Training: 2021-03-15 16:53:45,046-[agedb_30][52000]Accuracy-Highest: 0.97800 Training: 2021-03-15 16:54:10,915-Speed 227.18 samples/sec Loss 4.6776 Epoch: 10 Global Step: 52050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:54:36,281-Speed 2018.55 samples/sec Loss 4.6655 Epoch: 10 Global Step: 52100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:55:01,928-Speed 1996.41 samples/sec Loss 4.5767 Epoch: 10 Global Step: 52150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:55:27,926-Speed 1969.47 samples/sec Loss 4.5416 Epoch: 10 Global Step: 52200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:55:53,532-Speed 1999.61 samples/sec Loss 4.5866 Epoch: 10 Global Step: 52250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:56:19,021-Speed 2008.75 samples/sec Loss 4.6127 Epoch: 10 Global Step: 52300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:56:44,755-Speed 1989.70 samples/sec Loss 4.5053 Epoch: 10 Global Step: 52350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:57:10,358-Speed 1999.88 samples/sec Loss 4.4777 Epoch: 10 Global Step: 52400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:57:35,910-Speed 2003.82 samples/sec Loss 4.5098 Epoch: 10 Global Step: 52450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:58:01,549-Speed 1997.06 samples/sec Loss 4.4008 Epoch: 10 Global Step: 52500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:58:27,203-Speed 1995.84 samples/sec Loss 4.4380 Epoch: 10 Global Step: 52550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:58:53,204-Speed 1969.26 samples/sec Loss 4.3982 Epoch: 10 Global Step: 52600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:59:19,368-Speed 1956.92 samples/sec Loss 4.3243 Epoch: 10 Global Step: 52650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 16:59:45,112-Speed 1988.91 samples/sec Loss 4.3548 Epoch: 10 Global Step: 52700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:00:10,727-Speed 1998.85 samples/sec Loss 4.3831 Epoch: 10 Global Step: 52750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:00:36,220-Speed 2008.52 samples/sec Loss 4.3190 Epoch: 10 Global Step: 52800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:01:02,027-Speed 1984.08 samples/sec Loss 4.2981 Epoch: 10 Global Step: 52850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:01:28,103-Speed 1963.57 samples/sec Loss 4.3119 Epoch: 10 Global Step: 52900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:01:54,160-Speed 1965.03 samples/sec Loss 4.2733 Epoch: 10 Global Step: 52950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:02:19,697-Speed 2004.95 samples/sec Loss 4.2752 Epoch: 10 Global Step: 53000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:02:45,623-Speed 1974.89 samples/sec Loss 4.2641 Epoch: 10 Global Step: 53050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:03:11,558-Speed 1974.23 samples/sec Loss 4.2560 Epoch: 10 Global Step: 53100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:03:37,763-Speed 1953.92 samples/sec Loss 4.2449 Epoch: 10 Global Step: 53150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:04:03,673-Speed 1976.06 samples/sec Loss 4.1952 Epoch: 10 Global Step: 53200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:04:29,963-Speed 1947.62 samples/sec Loss 4.1876 Epoch: 10 Global Step: 53250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:04:56,229-Speed 1949.32 samples/sec Loss 4.1829 Epoch: 10 Global Step: 53300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:05:21,907-Speed 1993.98 samples/sec Loss 4.1569 Epoch: 10 Global Step: 53350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:05:47,864-Speed 1972.59 samples/sec Loss 4.1934 Epoch: 10 Global Step: 53400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:06:13,916-Speed 1965.34 samples/sec Loss 4.1653 Epoch: 10 Global Step: 53450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:06:39,536-Speed 1998.50 samples/sec Loss 4.1670 Epoch: 10 Global Step: 53500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:07:05,249-Speed 1991.36 samples/sec Loss 4.1860 Epoch: 10 Global Step: 53550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:07:30,728-Speed 2009.54 samples/sec Loss 4.1308 Epoch: 10 Global Step: 53600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:07:56,637-Speed 1976.17 samples/sec Loss 4.1498 Epoch: 10 Global Step: 53650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:08:22,923-Speed 1947.86 samples/sec Loss 4.1247 Epoch: 10 Global Step: 53700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:08:48,901-Speed 1971.00 samples/sec Loss 4.0967 Epoch: 10 Global Step: 53750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:09:14,759-Speed 1980.24 samples/sec Loss 4.1219 Epoch: 10 Global Step: 53800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:09:40,303-Speed 2004.42 samples/sec Loss 4.1246 Epoch: 10 Global Step: 53850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:10:06,298-Speed 1969.67 samples/sec Loss 4.1330 Epoch: 10 Global Step: 53900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:10:31,867-Speed 2002.61 samples/sec Loss 4.1544 Epoch: 10 Global Step: 53950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:10:57,832-Speed 1972.01 samples/sec Loss 4.1161 Epoch: 10 Global Step: 54000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:12:01,729-[lfw][54000]XNorm: 22.259048 Training: 2021-03-15 17:12:01,729-[lfw][54000]Accuracy-Flip: 0.99800+-0.00208 Training: 2021-03-15 17:12:01,730-[lfw][54000]Accuracy-Highest: 0.99800 Training: 2021-03-15 17:13:16,604-[cfp_fp][54000]XNorm: 20.209106 Training: 2021-03-15 17:13:16,605-[cfp_fp][54000]Accuracy-Flip: 0.98557+-0.00401 Training: 2021-03-15 17:13:16,605-[cfp_fp][54000]Accuracy-Highest: 0.98657 Training: 2021-03-15 17:14:20,012-[agedb_30][54000]XNorm: 22.417885 Training: 2021-03-15 17:14:20,012-[agedb_30][54000]Accuracy-Flip: 0.97800+-0.00623 Training: 2021-03-15 17:14:20,012-[agedb_30][54000]Accuracy-Highest: 0.97800 Training: 2021-03-15 17:14:45,466-Speed 224.92 samples/sec Loss 4.1272 Epoch: 10 Global Step: 54050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:15:11,246-Speed 1986.07 samples/sec Loss 4.0944 Epoch: 10 Global Step: 54100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:15:36,620-Speed 2017.96 samples/sec Loss 4.0932 Epoch: 10 Global Step: 54150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:16:02,319-Speed 1992.30 samples/sec Loss 4.0924 Epoch: 10 Global Step: 54200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:16:28,019-Speed 1992.27 samples/sec Loss 4.0475 Epoch: 10 Global Step: 54250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:16:54,156-Speed 1959.01 samples/sec Loss 4.0753 Epoch: 10 Global Step: 54300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:17:19,960-Speed 1984.25 samples/sec Loss 4.0513 Epoch: 10 Global Step: 54350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:17:45,934-Speed 1971.26 samples/sec Loss 4.1065 Epoch: 10 Global Step: 54400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:18:11,735-Speed 1984.57 samples/sec Loss 4.0448 Epoch: 10 Global Step: 54450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:18:37,212-Speed 2009.70 samples/sec Loss 4.0655 Epoch: 10 Global Step: 54500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:19:03,033-Speed 1982.89 samples/sec Loss 4.0763 Epoch: 10 Global Step: 54550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:19:28,412-Speed 2017.49 samples/sec Loss 4.0610 Epoch: 10 Global Step: 54600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:19:53,981-Speed 2002.51 samples/sec Loss 4.0754 Epoch: 10 Global Step: 54650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:20:19,809-Speed 1982.44 samples/sec Loss 4.0843 Epoch: 10 Global Step: 54700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:20:45,991-Speed 1955.64 samples/sec Loss 4.0060 Epoch: 10 Global Step: 54750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:21:11,583-Speed 2000.66 samples/sec Loss 4.0929 Epoch: 10 Global Step: 54800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:21:42,012-Speed 1682.64 samples/sec Loss 3.2973 Epoch: 11 Global Step: 54850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:22:08,694-Speed 1918.98 samples/sec Loss 3.3056 Epoch: 11 Global Step: 54900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:22:34,172-Speed 2009.72 samples/sec Loss 3.3220 Epoch: 11 Global Step: 54950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:22:59,648-Speed 2009.77 samples/sec Loss 3.3321 Epoch: 11 Global Step: 55000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:23:25,101-Speed 2011.69 samples/sec Loss 3.3731 Epoch: 11 Global Step: 55050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:23:51,173-Speed 1963.86 samples/sec Loss 3.3982 Epoch: 11 Global Step: 55100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:24:17,297-Speed 1959.94 samples/sec Loss 3.4114 Epoch: 11 Global Step: 55150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:24:42,994-Speed 1992.55 samples/sec Loss 3.4753 Epoch: 11 Global Step: 55200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:25:08,534-Speed 2004.78 samples/sec Loss 3.4914 Epoch: 11 Global Step: 55250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:25:34,570-Speed 1966.54 samples/sec Loss 3.5378 Epoch: 11 Global Step: 55300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:26:00,230-Speed 1995.44 samples/sec Loss 3.5291 Epoch: 11 Global Step: 55350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:26:25,743-Speed 2006.86 samples/sec Loss 3.5643 Epoch: 11 Global Step: 55400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:26:51,212-Speed 2010.38 samples/sec Loss 3.5895 Epoch: 11 Global Step: 55450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:27:16,975-Speed 1987.35 samples/sec Loss 3.5787 Epoch: 11 Global Step: 55500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:27:43,030-Speed 1965.16 samples/sec Loss 3.5595 Epoch: 11 Global Step: 55550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:28:09,020-Speed 1970.05 samples/sec Loss 3.6350 Epoch: 11 Global Step: 55600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:28:34,559-Speed 2004.88 samples/sec Loss 3.6308 Epoch: 11 Global Step: 55650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:29:00,368-Speed 1983.83 samples/sec Loss 3.6072 Epoch: 11 Global Step: 55700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:29:26,155-Speed 1985.61 samples/sec Loss 3.7011 Epoch: 11 Global Step: 55750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:29:52,353-Speed 1954.37 samples/sec Loss 3.6397 Epoch: 11 Global Step: 55800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:30:18,738-Speed 1940.62 samples/sec Loss 3.7170 Epoch: 11 Global Step: 55850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:30:45,360-Speed 1923.25 samples/sec Loss 3.7416 Epoch: 11 Global Step: 55900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:31:11,496-Speed 1959.06 samples/sec Loss 3.7769 Epoch: 11 Global Step: 55950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:31:37,582-Speed 1962.80 samples/sec Loss 3.7240 Epoch: 11 Global Step: 56000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:32:40,536-[lfw][56000]XNorm: 23.414764 Training: 2021-03-15 17:32:40,536-[lfw][56000]Accuracy-Flip: 0.99783+-0.00236 Training: 2021-03-15 17:32:40,536-[lfw][56000]Accuracy-Highest: 0.99800 Training: 2021-03-15 17:33:53,159-[cfp_fp][56000]XNorm: 21.436731 Training: 2021-03-15 17:33:53,159-[cfp_fp][56000]Accuracy-Flip: 0.98543+-0.00343 Training: 2021-03-15 17:33:53,159-[cfp_fp][56000]Accuracy-Highest: 0.98657 Training: 2021-03-15 17:34:56,186-[agedb_30][56000]XNorm: 23.373412 Training: 2021-03-15 17:34:56,186-[agedb_30][56000]Accuracy-Flip: 0.97900+-0.00684 Training: 2021-03-15 17:34:56,186-[agedb_30][56000]Accuracy-Highest: 0.97900 Training: 2021-03-15 17:35:21,996-Speed 228.15 samples/sec Loss 3.8110 Epoch: 11 Global Step: 56050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:35:48,338-Speed 1943.77 samples/sec Loss 3.7350 Epoch: 11 Global Step: 56100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:36:14,348-Speed 1968.51 samples/sec Loss 3.7866 Epoch: 11 Global Step: 56150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:36:40,829-Speed 1933.49 samples/sec Loss 3.8282 Epoch: 11 Global Step: 56200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:37:07,012-Speed 1955.58 samples/sec Loss 3.8443 Epoch: 11 Global Step: 56250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:37:33,275-Speed 1949.58 samples/sec Loss 3.8018 Epoch: 11 Global Step: 56300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:37:59,515-Speed 1951.25 samples/sec Loss 3.8336 Epoch: 11 Global Step: 56350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:38:25,701-Speed 1955.30 samples/sec Loss 3.8965 Epoch: 11 Global Step: 56400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:38:52,072-Speed 1941.57 samples/sec Loss 3.9223 Epoch: 11 Global Step: 56450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:39:18,867-Speed 1910.88 samples/sec Loss 3.8969 Epoch: 11 Global Step: 56500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:39:45,661-Speed 1911.10 samples/sec Loss 3.8863 Epoch: 11 Global Step: 56550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:40:12,018-Speed 1942.61 samples/sec Loss 3.8691 Epoch: 11 Global Step: 56600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:40:37,837-Speed 1983.09 samples/sec Loss 3.9117 Epoch: 11 Global Step: 56650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:41:04,170-Speed 1944.41 samples/sec Loss 3.9095 Epoch: 11 Global Step: 56700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:41:30,549-Speed 1941.02 samples/sec Loss 3.8840 Epoch: 11 Global Step: 56750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:41:56,666-Speed 1960.49 samples/sec Loss 3.9612 Epoch: 11 Global Step: 56800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:42:23,340-Speed 1919.62 samples/sec Loss 3.9336 Epoch: 11 Global Step: 56850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:42:49,603-Speed 1949.56 samples/sec Loss 3.8805 Epoch: 11 Global Step: 56900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:43:15,857-Speed 1950.24 samples/sec Loss 3.9873 Epoch: 11 Global Step: 56950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:43:42,188-Speed 1944.59 samples/sec Loss 3.9816 Epoch: 11 Global Step: 57000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:44:08,657-Speed 1934.41 samples/sec Loss 3.9796 Epoch: 11 Global Step: 57050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:44:34,952-Speed 1947.16 samples/sec Loss 4.0112 Epoch: 11 Global Step: 57100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:45:01,230-Speed 1948.46 samples/sec Loss 3.9520 Epoch: 11 Global Step: 57150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:45:27,183-Speed 1972.87 samples/sec Loss 4.0233 Epoch: 11 Global Step: 57200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:45:53,237-Speed 1965.21 samples/sec Loss 4.0630 Epoch: 11 Global Step: 57250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:46:19,469-Speed 1951.88 samples/sec Loss 4.0021 Epoch: 11 Global Step: 57300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:46:45,998-Speed 1930.05 samples/sec Loss 4.0354 Epoch: 11 Global Step: 57350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:47:12,421-Speed 1937.75 samples/sec Loss 4.0101 Epoch: 11 Global Step: 57400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:47:38,506-Speed 1962.92 samples/sec Loss 4.0858 Epoch: 11 Global Step: 57450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:48:05,330-Speed 1908.76 samples/sec Loss 4.0304 Epoch: 11 Global Step: 57500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:48:31,592-Speed 1949.67 samples/sec Loss 4.0318 Epoch: 11 Global Step: 57550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:48:58,411-Speed 1909.21 samples/sec Loss 4.0809 Epoch: 11 Global Step: 57600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:49:25,062-Speed 1921.14 samples/sec Loss 4.1176 Epoch: 11 Global Step: 57650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:49:51,425-Speed 1942.20 samples/sec Loss 4.0973 Epoch: 11 Global Step: 57700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:50:18,009-Speed 1926.05 samples/sec Loss 4.0797 Epoch: 11 Global Step: 57750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:50:44,499-Speed 1932.81 samples/sec Loss 4.1030 Epoch: 11 Global Step: 57800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:51:10,555-Speed 1965.10 samples/sec Loss 4.1002 Epoch: 11 Global Step: 57850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:51:36,907-Speed 1943.00 samples/sec Loss 4.0670 Epoch: 11 Global Step: 57900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:52:03,458-Speed 1928.40 samples/sec Loss 4.0991 Epoch: 11 Global Step: 57950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:52:29,844-Speed 1940.52 samples/sec Loss 4.1358 Epoch: 11 Global Step: 58000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:53:34,008-[lfw][58000]XNorm: 22.237516 Training: 2021-03-15 17:53:34,009-[lfw][58000]Accuracy-Flip: 0.99783+-0.00211 Training: 2021-03-15 17:53:34,009-[lfw][58000]Accuracy-Highest: 0.99800 Training: 2021-03-15 17:54:46,404-[cfp_fp][58000]XNorm: 20.330965 Training: 2021-03-15 17:54:46,404-[cfp_fp][58000]Accuracy-Flip: 0.98443+-0.00380 Training: 2021-03-15 17:54:46,404-[cfp_fp][58000]Accuracy-Highest: 0.98657 Training: 2021-03-15 17:55:50,607-[agedb_30][58000]XNorm: 22.414971 Training: 2021-03-15 17:55:50,607-[agedb_30][58000]Accuracy-Flip: 0.98067+-0.00750 Training: 2021-03-15 17:55:50,608-[agedb_30][58000]Accuracy-Highest: 0.98067 Training: 2021-03-15 17:56:17,283-Speed 225.12 samples/sec Loss 4.1334 Epoch: 11 Global Step: 58050 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:56:43,543-Speed 1949.85 samples/sec Loss 4.1379 Epoch: 11 Global Step: 58100 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:57:09,857-Speed 1945.75 samples/sec Loss 4.1102 Epoch: 11 Global Step: 58150 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:57:36,489-Speed 1922.60 samples/sec Loss 4.1415 Epoch: 11 Global Step: 58200 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:58:02,552-Speed 1964.51 samples/sec Loss 4.1688 Epoch: 11 Global Step: 58250 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:58:28,561-Speed 1968.58 samples/sec Loss 4.1281 Epoch: 11 Global Step: 58300 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:58:54,847-Speed 1947.94 samples/sec Loss 4.1416 Epoch: 11 Global Step: 58350 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:59:21,020-Speed 1956.38 samples/sec Loss 4.1541 Epoch: 11 Global Step: 58400 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 17:59:47,411-Speed 1940.10 samples/sec Loss 4.1404 Epoch: 11 Global Step: 58450 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:00:13,720-Speed 1946.37 samples/sec Loss 4.2358 Epoch: 11 Global Step: 58500 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:00:40,490-Speed 1912.76 samples/sec Loss 4.1807 Epoch: 11 Global Step: 58550 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:01:06,807-Speed 1945.75 samples/sec Loss 4.1688 Epoch: 11 Global Step: 58600 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:01:32,727-Speed 1975.35 samples/sec Loss 4.1710 Epoch: 11 Global Step: 58650 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:01:58,778-Speed 1965.45 samples/sec Loss 4.1863 Epoch: 11 Global Step: 58700 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:02:25,113-Speed 1944.27 samples/sec Loss 4.2222 Epoch: 11 Global Step: 58750 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:02:51,192-Speed 1963.29 samples/sec Loss 4.1529 Epoch: 11 Global Step: 58800 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:03:17,564-Speed 1941.53 samples/sec Loss 4.1983 Epoch: 11 Global Step: 58850 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:03:44,059-Speed 1932.53 samples/sec Loss 4.1719 Epoch: 11 Global Step: 58900 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:04:10,655-Speed 1925.16 samples/sec Loss 4.2301 Epoch: 11 Global Step: 58950 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:04:36,791-Speed 1959.07 samples/sec Loss 4.2547 Epoch: 11 Global Step: 59000 Fp16 Grad Scale: 16384 Required: 10 hours Training: 2021-03-15 18:05:03,187-Speed 1939.73 samples/sec Loss 4.2497 Epoch: 11 Global Step: 59050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:05:29,492-Speed 1946.47 samples/sec Loss 4.1796 Epoch: 11 Global Step: 59100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:05:56,272-Speed 1911.91 samples/sec Loss 4.2331 Epoch: 11 Global Step: 59150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:06:22,206-Speed 1974.37 samples/sec Loss 4.3122 Epoch: 11 Global Step: 59200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:06:48,588-Speed 1940.75 samples/sec Loss 4.2472 Epoch: 11 Global Step: 59250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:07:14,981-Speed 1939.98 samples/sec Loss 4.2812 Epoch: 11 Global Step: 59300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:07:41,301-Speed 1945.38 samples/sec Loss 4.2667 Epoch: 11 Global Step: 59350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:08:08,175-Speed 1905.22 samples/sec Loss 4.2777 Epoch: 11 Global Step: 59400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:08:34,215-Speed 1966.30 samples/sec Loss 4.2568 Epoch: 11 Global Step: 59450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:09:00,467-Speed 1950.35 samples/sec Loss 4.2607 Epoch: 11 Global Step: 59500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:09:26,632-Speed 1956.89 samples/sec Loss 4.2411 Epoch: 11 Global Step: 59550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:09:53,254-Speed 1923.28 samples/sec Loss 4.2735 Epoch: 11 Global Step: 59600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:10:19,755-Speed 1932.02 samples/sec Loss 4.2720 Epoch: 11 Global Step: 59650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:10:45,991-Speed 1951.58 samples/sec Loss 4.3090 Epoch: 11 Global Step: 59700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:11:12,692-Speed 1917.61 samples/sec Loss 4.3026 Epoch: 11 Global Step: 59750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:11:44,822-Speed 1593.63 samples/sec Loss 3.9975 Epoch: 12 Global Step: 59800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:12:11,296-Speed 1934.00 samples/sec Loss 3.4884 Epoch: 12 Global Step: 59850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:12:37,474-Speed 1955.92 samples/sec Loss 3.5676 Epoch: 12 Global Step: 59900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:13:04,186-Speed 1916.97 samples/sec Loss 3.5627 Epoch: 12 Global Step: 59950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:13:30,403-Speed 1952.96 samples/sec Loss 3.5907 Epoch: 12 Global Step: 60000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:14:34,535-[lfw][60000]XNorm: 22.667547 Training: 2021-03-15 18:14:34,536-[lfw][60000]Accuracy-Flip: 0.99800+-0.00277 Training: 2021-03-15 18:14:34,536-[lfw][60000]Accuracy-Highest: 0.99800 Training: 2021-03-15 18:15:47,822-[cfp_fp][60000]XNorm: 20.389199 Training: 2021-03-15 18:15:47,822-[cfp_fp][60000]Accuracy-Flip: 0.98571+-0.00356 Training: 2021-03-15 18:15:47,822-[cfp_fp][60000]Accuracy-Highest: 0.98657 Training: 2021-03-15 18:16:50,027-[agedb_30][60000]XNorm: 22.913784 Training: 2021-03-15 18:16:50,027-[agedb_30][60000]Accuracy-Flip: 0.98000+-0.00782 Training: 2021-03-15 18:16:50,029-[agedb_30][60000]Accuracy-Highest: 0.98067 Training: 2021-03-15 18:17:16,025-Speed 226.93 samples/sec Loss 3.6890 Epoch: 12 Global Step: 60050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:17:41,976-Speed 1972.97 samples/sec Loss 3.6780 Epoch: 12 Global Step: 60100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:18:08,496-Speed 1930.71 samples/sec Loss 3.7106 Epoch: 12 Global Step: 60150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:18:34,532-Speed 1966.61 samples/sec Loss 3.7588 Epoch: 12 Global Step: 60200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:19:00,928-Speed 1939.74 samples/sec Loss 3.7736 Epoch: 12 Global Step: 60250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:19:27,433-Speed 1931.80 samples/sec Loss 3.7224 Epoch: 12 Global Step: 60300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:19:53,621-Speed 1955.17 samples/sec Loss 3.8567 Epoch: 12 Global Step: 60350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:20:20,245-Speed 1923.17 samples/sec Loss 3.8193 Epoch: 12 Global Step: 60400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:20:46,757-Speed 1931.24 samples/sec Loss 3.8308 Epoch: 12 Global Step: 60450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:21:13,146-Speed 1940.24 samples/sec Loss 3.8423 Epoch: 12 Global Step: 60500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:21:39,681-Speed 1929.61 samples/sec Loss 3.8982 Epoch: 12 Global Step: 60550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:22:06,225-Speed 1928.97 samples/sec Loss 3.9480 Epoch: 12 Global Step: 60600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:22:32,335-Speed 1961.00 samples/sec Loss 3.9358 Epoch: 12 Global Step: 60650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:22:58,733-Speed 1939.56 samples/sec Loss 3.9378 Epoch: 12 Global Step: 60700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:23:25,362-Speed 1922.91 samples/sec Loss 3.9715 Epoch: 12 Global Step: 60750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:23:52,020-Speed 1920.65 samples/sec Loss 3.9900 Epoch: 12 Global Step: 60800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:24:18,630-Speed 1924.20 samples/sec Loss 3.9863 Epoch: 12 Global Step: 60850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:24:45,091-Speed 1934.99 samples/sec Loss 4.0508 Epoch: 12 Global Step: 60900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:25:11,369-Speed 1948.51 samples/sec Loss 4.0066 Epoch: 12 Global Step: 60950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:25:37,440-Speed 1963.90 samples/sec Loss 4.0948 Epoch: 12 Global Step: 61000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:26:04,432-Speed 1896.94 samples/sec Loss 4.0683 Epoch: 12 Global Step: 61050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:26:30,561-Speed 1959.63 samples/sec Loss 4.0925 Epoch: 12 Global Step: 61100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:26:56,797-Speed 1951.58 samples/sec Loss 4.0688 Epoch: 12 Global Step: 61150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:27:22,851-Speed 1965.20 samples/sec Loss 4.0795 Epoch: 12 Global Step: 61200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:27:49,036-Speed 1955.44 samples/sec Loss 4.1392 Epoch: 12 Global Step: 61250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:28:14,878-Speed 1981.34 samples/sec Loss 4.1528 Epoch: 12 Global Step: 61300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:28:41,574-Speed 1917.98 samples/sec Loss 4.1261 Epoch: 12 Global Step: 61350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:29:08,203-Speed 1922.77 samples/sec Loss 4.1264 Epoch: 12 Global Step: 61400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:29:35,471-Speed 1877.73 samples/sec Loss 4.1732 Epoch: 12 Global Step: 61450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:30:01,457-Speed 1970.41 samples/sec Loss 4.1705 Epoch: 12 Global Step: 61500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:30:27,912-Speed 1935.39 samples/sec Loss 4.1856 Epoch: 12 Global Step: 61550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:30:54,570-Speed 1920.68 samples/sec Loss 4.1268 Epoch: 12 Global Step: 61600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:31:20,603-Speed 1966.81 samples/sec Loss 4.2004 Epoch: 12 Global Step: 61650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:31:46,957-Speed 1942.86 samples/sec Loss 4.1518 Epoch: 12 Global Step: 61700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:32:13,551-Speed 1925.33 samples/sec Loss 4.2023 Epoch: 12 Global Step: 61750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:32:40,519-Speed 1898.64 samples/sec Loss 4.2001 Epoch: 12 Global Step: 61800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:33:06,791-Speed 1948.90 samples/sec Loss 4.1515 Epoch: 12 Global Step: 61850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:33:33,380-Speed 1925.64 samples/sec Loss 4.2112 Epoch: 12 Global Step: 61900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:33:59,735-Speed 1942.76 samples/sec Loss 4.2132 Epoch: 12 Global Step: 61950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:34:25,660-Speed 1974.98 samples/sec Loss 4.2170 Epoch: 12 Global Step: 62000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:35:28,563-[lfw][62000]XNorm: 22.712712 Training: 2021-03-15 18:35:28,563-[lfw][62000]Accuracy-Flip: 0.99783+-0.00211 Training: 2021-03-15 18:35:28,563-[lfw][62000]Accuracy-Highest: 0.99800 Training: 2021-03-15 18:36:41,359-[cfp_fp][62000]XNorm: 20.717737 Training: 2021-03-15 18:36:41,359-[cfp_fp][62000]Accuracy-Flip: 0.98643+-0.00224 Training: 2021-03-15 18:36:41,359-[cfp_fp][62000]Accuracy-Highest: 0.98657 Training: 2021-03-15 18:37:44,963-[agedb_30][62000]XNorm: 22.684769 Training: 2021-03-15 18:37:44,964-[agedb_30][62000]Accuracy-Flip: 0.97950+-0.00687 Training: 2021-03-15 18:37:44,964-[agedb_30][62000]Accuracy-Highest: 0.98067 Training: 2021-03-15 18:38:11,191-Speed 227.02 samples/sec Loss 4.2432 Epoch: 12 Global Step: 62050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:38:37,342-Speed 1957.89 samples/sec Loss 4.2595 Epoch: 12 Global Step: 62100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:39:03,633-Speed 1947.56 samples/sec Loss 4.2749 Epoch: 12 Global Step: 62150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:39:29,797-Speed 1956.94 samples/sec Loss 4.2962 Epoch: 12 Global Step: 62200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:39:55,974-Speed 1955.94 samples/sec Loss 4.3064 Epoch: 12 Global Step: 62250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:40:22,580-Speed 1924.45 samples/sec Loss 4.2919 Epoch: 12 Global Step: 62300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:40:49,006-Speed 1937.59 samples/sec Loss 4.3177 Epoch: 12 Global Step: 62350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:41:15,583-Speed 1926.56 samples/sec Loss 4.3177 Epoch: 12 Global Step: 62400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:41:42,242-Speed 1920.57 samples/sec Loss 4.2773 Epoch: 12 Global Step: 62450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:42:08,200-Speed 1972.51 samples/sec Loss 4.2241 Epoch: 12 Global Step: 62500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:42:34,129-Speed 1974.70 samples/sec Loss 4.3032 Epoch: 12 Global Step: 62550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:42:59,987-Speed 1980.24 samples/sec Loss 4.3049 Epoch: 12 Global Step: 62600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:43:26,405-Speed 1938.16 samples/sec Loss 4.3202 Epoch: 12 Global Step: 62650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:43:53,527-Speed 1887.84 samples/sec Loss 4.2920 Epoch: 12 Global Step: 62700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:44:20,043-Speed 1931.01 samples/sec Loss 4.3306 Epoch: 12 Global Step: 62750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:44:46,275-Speed 1951.85 samples/sec Loss 4.3417 Epoch: 12 Global Step: 62800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:45:12,885-Speed 1924.17 samples/sec Loss 4.3761 Epoch: 12 Global Step: 62850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:45:39,467-Speed 1926.22 samples/sec Loss 4.3118 Epoch: 12 Global Step: 62900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:46:06,030-Speed 1927.53 samples/sec Loss 4.3442 Epoch: 12 Global Step: 62950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:46:32,120-Speed 1962.53 samples/sec Loss 4.2833 Epoch: 12 Global Step: 63000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:46:58,794-Speed 1919.52 samples/sec Loss 4.3174 Epoch: 12 Global Step: 63050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:47:25,551-Speed 1913.61 samples/sec Loss 4.3617 Epoch: 12 Global Step: 63100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:47:51,935-Speed 1940.64 samples/sec Loss 4.3010 Epoch: 12 Global Step: 63150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:48:18,129-Speed 1954.75 samples/sec Loss 4.3071 Epoch: 12 Global Step: 63200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:48:44,629-Speed 1932.09 samples/sec Loss 4.2749 Epoch: 12 Global Step: 63250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:49:11,433-Speed 1910.26 samples/sec Loss 4.3126 Epoch: 12 Global Step: 63300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:49:37,443-Speed 1968.58 samples/sec Loss 4.3804 Epoch: 12 Global Step: 63350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:50:04,062-Speed 1923.45 samples/sec Loss 4.3414 Epoch: 12 Global Step: 63400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:50:30,868-Speed 1910.09 samples/sec Loss 4.3600 Epoch: 12 Global Step: 63450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:50:57,178-Speed 1946.08 samples/sec Loss 4.3429 Epoch: 12 Global Step: 63500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:51:23,417-Speed 1951.34 samples/sec Loss 4.3500 Epoch: 12 Global Step: 63550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:51:49,582-Speed 1956.88 samples/sec Loss 4.3101 Epoch: 12 Global Step: 63600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:52:15,857-Speed 1948.68 samples/sec Loss 4.3868 Epoch: 12 Global Step: 63650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:52:42,105-Speed 1950.73 samples/sec Loss 4.4392 Epoch: 12 Global Step: 63700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:53:08,588-Speed 1933.39 samples/sec Loss 4.3378 Epoch: 12 Global Step: 63750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:53:34,655-Speed 1964.22 samples/sec Loss 4.3850 Epoch: 12 Global Step: 63800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:54:01,258-Speed 1924.62 samples/sec Loss 4.3501 Epoch: 12 Global Step: 63850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:54:27,598-Speed 1943.91 samples/sec Loss 4.3878 Epoch: 12 Global Step: 63900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:54:54,447-Speed 1906.99 samples/sec Loss 4.3414 Epoch: 12 Global Step: 63950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:55:20,615-Speed 1956.63 samples/sec Loss 4.3649 Epoch: 12 Global Step: 64000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:56:23,614-[lfw][64000]XNorm: 24.383160 Training: 2021-03-15 18:56:23,614-[lfw][64000]Accuracy-Flip: 0.99800+-0.00221 Training: 2021-03-15 18:56:23,614-[lfw][64000]Accuracy-Highest: 0.99800 Training: 2021-03-15 18:57:36,089-[cfp_fp][64000]XNorm: 21.923103 Training: 2021-03-15 18:57:36,090-[cfp_fp][64000]Accuracy-Flip: 0.98600+-0.00210 Training: 2021-03-15 18:57:36,090-[cfp_fp][64000]Accuracy-Highest: 0.98657 Training: 2021-03-15 18:58:40,376-[agedb_30][64000]XNorm: 23.698406 Training: 2021-03-15 18:58:40,377-[agedb_30][64000]Accuracy-Flip: 0.97783+-0.00553 Training: 2021-03-15 18:58:40,377-[agedb_30][64000]Accuracy-Highest: 0.98067 Training: 2021-03-15 18:59:06,856-Speed 226.31 samples/sec Loss 4.3474 Epoch: 12 Global Step: 64050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:59:33,113-Speed 1949.98 samples/sec Loss 4.3387 Epoch: 12 Global Step: 64100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 18:59:59,362-Speed 1950.67 samples/sec Loss 4.3363 Epoch: 12 Global Step: 64150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:00:25,624-Speed 1949.60 samples/sec Loss 4.3919 Epoch: 12 Global Step: 64200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:00:51,871-Speed 1950.76 samples/sec Loss 4.3854 Epoch: 12 Global Step: 64250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:01:18,130-Speed 1949.87 samples/sec Loss 4.3192 Epoch: 12 Global Step: 64300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:01:44,818-Speed 1918.55 samples/sec Loss 4.3663 Epoch: 12 Global Step: 64350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:02:10,562-Speed 1988.88 samples/sec Loss 4.3842 Epoch: 12 Global Step: 64400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:02:36,756-Speed 1954.71 samples/sec Loss 4.3951 Epoch: 12 Global Step: 64450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:03:02,895-Speed 1958.80 samples/sec Loss 4.3948 Epoch: 12 Global Step: 64500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:03:29,190-Speed 1947.20 samples/sec Loss 4.3905 Epoch: 12 Global Step: 64550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:03:55,739-Speed 1928.59 samples/sec Loss 4.3809 Epoch: 12 Global Step: 64600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:04:22,163-Speed 1937.68 samples/sec Loss 4.3903 Epoch: 12 Global Step: 64650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:04:48,567-Speed 1939.16 samples/sec Loss 4.4045 Epoch: 12 Global Step: 64700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:05:15,262-Speed 1918.03 samples/sec Loss 4.3858 Epoch: 12 Global Step: 64750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:05:47,051-Speed 1610.66 samples/sec Loss 3.8926 Epoch: 13 Global Step: 64800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:06:13,184-Speed 1959.30 samples/sec Loss 3.6386 Epoch: 13 Global Step: 64850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:06:39,346-Speed 1957.08 samples/sec Loss 3.6889 Epoch: 13 Global Step: 64900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:07:06,137-Speed 1911.18 samples/sec Loss 3.7082 Epoch: 13 Global Step: 64950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:07:32,225-Speed 1962.64 samples/sec Loss 3.7196 Epoch: 13 Global Step: 65000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:07:59,031-Speed 1910.05 samples/sec Loss 3.7527 Epoch: 13 Global Step: 65050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:08:25,426-Speed 1939.78 samples/sec Loss 3.7910 Epoch: 13 Global Step: 65100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:08:52,155-Speed 1915.63 samples/sec Loss 3.8310 Epoch: 13 Global Step: 65150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:09:18,548-Speed 1939.91 samples/sec Loss 3.8539 Epoch: 13 Global Step: 65200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:09:44,694-Speed 1958.30 samples/sec Loss 3.8765 Epoch: 13 Global Step: 65250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:10:11,114-Speed 1938.03 samples/sec Loss 3.9091 Epoch: 13 Global Step: 65300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:10:37,332-Speed 1952.92 samples/sec Loss 3.9241 Epoch: 13 Global Step: 65350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:11:03,532-Speed 1954.29 samples/sec Loss 3.9691 Epoch: 13 Global Step: 65400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:11:29,482-Speed 1973.07 samples/sec Loss 3.9689 Epoch: 13 Global Step: 65450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:11:55,810-Speed 1944.77 samples/sec Loss 3.9945 Epoch: 13 Global Step: 65500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:12:22,329-Speed 1930.74 samples/sec Loss 4.0394 Epoch: 13 Global Step: 65550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:12:48,270-Speed 1973.77 samples/sec Loss 4.0624 Epoch: 13 Global Step: 65600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:13:14,390-Speed 1960.25 samples/sec Loss 4.0402 Epoch: 13 Global Step: 65650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:13:40,757-Speed 1941.89 samples/sec Loss 4.0714 Epoch: 13 Global Step: 65700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:14:07,166-Speed 1938.95 samples/sec Loss 4.0689 Epoch: 13 Global Step: 65750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:14:33,727-Speed 1927.74 samples/sec Loss 4.1046 Epoch: 13 Global Step: 65800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:14:59,707-Speed 1970.75 samples/sec Loss 4.0917 Epoch: 13 Global Step: 65850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:15:26,044-Speed 1944.09 samples/sec Loss 4.0651 Epoch: 13 Global Step: 65900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:15:53,345-Speed 1875.46 samples/sec Loss 4.1459 Epoch: 13 Global Step: 65950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:16:19,968-Speed 1923.22 samples/sec Loss 4.1489 Epoch: 13 Global Step: 66000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:17:22,230-[lfw][66000]XNorm: 24.396944 Training: 2021-03-15 19:17:22,230-[lfw][66000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-15 19:17:22,230-[lfw][66000]Accuracy-Highest: 0.99833 Training: 2021-03-15 19:18:36,446-[cfp_fp][66000]XNorm: 22.190780 Training: 2021-03-15 19:18:36,447-[cfp_fp][66000]Accuracy-Flip: 0.98429+-0.00389 Training: 2021-03-15 19:18:36,447-[cfp_fp][66000]Accuracy-Highest: 0.98657 Training: 2021-03-15 19:19:41,938-[agedb_30][66000]XNorm: 23.994527 Training: 2021-03-15 19:19:41,938-[agedb_30][66000]Accuracy-Flip: 0.97967+-0.00640 Training: 2021-03-15 19:19:41,938-[agedb_30][66000]Accuracy-Highest: 0.98067 Training: 2021-03-15 19:20:08,190-Speed 224.34 samples/sec Loss 4.1443 Epoch: 13 Global Step: 66050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:20:34,630-Speed 1936.55 samples/sec Loss 4.1519 Epoch: 13 Global Step: 66100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:21:01,040-Speed 1938.69 samples/sec Loss 4.1996 Epoch: 13 Global Step: 66150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:21:27,132-Speed 1962.38 samples/sec Loss 4.1836 Epoch: 13 Global Step: 66200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:21:53,873-Speed 1914.75 samples/sec Loss 4.2089 Epoch: 13 Global Step: 66250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:22:20,136-Speed 1949.54 samples/sec Loss 4.1549 Epoch: 13 Global Step: 66300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:22:46,405-Speed 1949.11 samples/sec Loss 4.2004 Epoch: 13 Global Step: 66350 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:23:13,177-Speed 1912.50 samples/sec Loss 4.2297 Epoch: 13 Global Step: 66400 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:23:39,698-Speed 1930.59 samples/sec Loss 4.2512 Epoch: 13 Global Step: 66450 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:24:06,416-Speed 1916.37 samples/sec Loss 4.2356 Epoch: 13 Global Step: 66500 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:24:32,747-Speed 1944.57 samples/sec Loss 4.2073 Epoch: 13 Global Step: 66550 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:24:59,034-Speed 1947.77 samples/sec Loss 4.2758 Epoch: 13 Global Step: 66600 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:25:25,269-Speed 1951.66 samples/sec Loss 4.2899 Epoch: 13 Global Step: 66650 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:25:51,750-Speed 1933.56 samples/sec Loss 4.2264 Epoch: 13 Global Step: 66700 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:26:18,103-Speed 1942.94 samples/sec Loss 4.2337 Epoch: 13 Global Step: 66750 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:26:44,306-Speed 1954.00 samples/sec Loss 4.2487 Epoch: 13 Global Step: 66800 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:27:10,443-Speed 1958.97 samples/sec Loss 4.2191 Epoch: 13 Global Step: 66850 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:27:36,741-Speed 1947.02 samples/sec Loss 4.2030 Epoch: 13 Global Step: 66900 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:28:03,093-Speed 1942.96 samples/sec Loss 4.2366 Epoch: 13 Global Step: 66950 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:28:29,055-Speed 1972.22 samples/sec Loss 4.2845 Epoch: 13 Global Step: 67000 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:28:55,428-Speed 1941.42 samples/sec Loss 4.2748 Epoch: 13 Global Step: 67050 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:29:21,812-Speed 1940.63 samples/sec Loss 4.2797 Epoch: 13 Global Step: 67100 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:29:48,281-Speed 1934.38 samples/sec Loss 4.2488 Epoch: 13 Global Step: 67150 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:30:14,722-Speed 1936.50 samples/sec Loss 4.3064 Epoch: 13 Global Step: 67200 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:30:40,796-Speed 1963.72 samples/sec Loss 4.3327 Epoch: 13 Global Step: 67250 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:31:07,349-Speed 1928.27 samples/sec Loss 4.3158 Epoch: 13 Global Step: 67300 Fp16 Grad Scale: 16384 Required: 9 hours Training: 2021-03-15 19:31:33,533-Speed 1955.47 samples/sec Loss 4.2965 Epoch: 13 Global Step: 67350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:31:59,911-Speed 1941.04 samples/sec Loss 4.3217 Epoch: 13 Global Step: 67400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:32:26,415-Speed 1931.88 samples/sec Loss 4.3089 Epoch: 13 Global Step: 67450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:32:52,752-Speed 1944.08 samples/sec Loss 4.3045 Epoch: 13 Global Step: 67500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:33:19,594-Speed 1907.51 samples/sec Loss 4.2887 Epoch: 13 Global Step: 67550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:33:45,919-Speed 1944.96 samples/sec Loss 4.3175 Epoch: 13 Global Step: 67600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:34:12,280-Speed 1942.29 samples/sec Loss 4.3678 Epoch: 13 Global Step: 67650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:34:39,189-Speed 1902.79 samples/sec Loss 4.3365 Epoch: 13 Global Step: 67700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:35:05,422-Speed 1951.80 samples/sec Loss 4.3753 Epoch: 13 Global Step: 67750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:35:31,707-Speed 1947.96 samples/sec Loss 4.3095 Epoch: 13 Global Step: 67800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:35:58,188-Speed 1933.46 samples/sec Loss 4.3300 Epoch: 13 Global Step: 67850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:36:24,628-Speed 1936.54 samples/sec Loss 4.3006 Epoch: 13 Global Step: 67900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:36:51,316-Speed 1918.66 samples/sec Loss 4.3411 Epoch: 13 Global Step: 67950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:37:17,535-Speed 1952.87 samples/sec Loss 4.3749 Epoch: 13 Global Step: 68000 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:38:22,039-[lfw][68000]XNorm: 22.129532 Training: 2021-03-15 19:38:22,039-[lfw][68000]Accuracy-Flip: 0.99817+-0.00174 Training: 2021-03-15 19:38:22,039-[lfw][68000]Accuracy-Highest: 0.99833 Training: 2021-03-15 19:39:37,135-[cfp_fp][68000]XNorm: 20.444580 Training: 2021-03-15 19:39:37,135-[cfp_fp][68000]Accuracy-Flip: 0.98514+-0.00374 Training: 2021-03-15 19:39:37,135-[cfp_fp][68000]Accuracy-Highest: 0.98657 Training: 2021-03-15 19:40:42,141-[agedb_30][68000]XNorm: 22.211393 Training: 2021-03-15 19:40:42,141-[agedb_30][68000]Accuracy-Flip: 0.97900+-0.00786 Training: 2021-03-15 19:40:42,141-[agedb_30][68000]Accuracy-Highest: 0.98067 Training: 2021-03-15 19:41:08,358-Speed 221.82 samples/sec Loss 4.3459 Epoch: 13 Global Step: 68050 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:41:35,560-Speed 1882.27 samples/sec Loss 4.2991 Epoch: 13 Global Step: 68100 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:42:01,550-Speed 1970.06 samples/sec Loss 4.3103 Epoch: 13 Global Step: 68150 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:42:27,974-Speed 1937.64 samples/sec Loss 4.3764 Epoch: 13 Global Step: 68200 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:42:54,211-Speed 1951.55 samples/sec Loss 4.3267 Epoch: 13 Global Step: 68250 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:43:20,728-Speed 1930.89 samples/sec Loss 4.3270 Epoch: 13 Global Step: 68300 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:43:47,005-Speed 1948.48 samples/sec Loss 4.3578 Epoch: 13 Global Step: 68350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:44:12,572-Speed 2002.71 samples/sec Loss 4.3154 Epoch: 13 Global Step: 68400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:44:38,961-Speed 1940.24 samples/sec Loss 4.3879 Epoch: 13 Global Step: 68450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:45:05,285-Speed 1945.04 samples/sec Loss 4.3500 Epoch: 13 Global Step: 68500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:45:31,757-Speed 1934.21 samples/sec Loss 4.3440 Epoch: 13 Global Step: 68550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:45:57,893-Speed 1959.00 samples/sec Loss 4.4081 Epoch: 13 Global Step: 68600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:46:23,958-Speed 1964.50 samples/sec Loss 4.3368 Epoch: 13 Global Step: 68650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:46:50,235-Speed 1948.52 samples/sec Loss 4.3416 Epoch: 13 Global Step: 68700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:47:16,508-Speed 1948.84 samples/sec Loss 4.3223 Epoch: 13 Global Step: 68750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:47:42,780-Speed 1948.93 samples/sec Loss 4.3733 Epoch: 13 Global Step: 68800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:48:08,623-Speed 1981.27 samples/sec Loss 4.3622 Epoch: 13 Global Step: 68850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:48:34,989-Speed 1941.93 samples/sec Loss 4.3520 Epoch: 13 Global Step: 68900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:49:01,190-Speed 1954.20 samples/sec Loss 4.3622 Epoch: 13 Global Step: 68950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:49:27,373-Speed 1955.49 samples/sec Loss 4.3834 Epoch: 13 Global Step: 69000 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:49:53,404-Speed 1967.01 samples/sec Loss 4.3091 Epoch: 13 Global Step: 69050 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:50:19,629-Speed 1952.37 samples/sec Loss 4.3658 Epoch: 13 Global Step: 69100 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:50:46,055-Speed 1937.58 samples/sec Loss 4.3899 Epoch: 13 Global Step: 69150 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:51:12,805-Speed 1914.03 samples/sec Loss 4.3422 Epoch: 13 Global Step: 69200 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:51:38,958-Speed 1957.95 samples/sec Loss 4.4126 Epoch: 13 Global Step: 69250 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:52:05,069-Speed 1960.92 samples/sec Loss 4.3444 Epoch: 13 Global Step: 69300 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:52:31,857-Speed 1911.38 samples/sec Loss 4.4198 Epoch: 13 Global Step: 69350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:52:58,489-Speed 1922.54 samples/sec Loss 4.3901 Epoch: 13 Global Step: 69400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:53:25,353-Speed 1905.93 samples/sec Loss 4.3357 Epoch: 13 Global Step: 69450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:53:52,083-Speed 1915.53 samples/sec Loss 4.3429 Epoch: 13 Global Step: 69500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:54:18,733-Speed 1921.23 samples/sec Loss 4.4331 Epoch: 13 Global Step: 69550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:54:45,349-Speed 1923.71 samples/sec Loss 4.3504 Epoch: 13 Global Step: 69600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:55:11,617-Speed 1949.23 samples/sec Loss 4.3749 Epoch: 13 Global Step: 69650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:55:38,558-Speed 1900.47 samples/sec Loss 4.3931 Epoch: 13 Global Step: 69700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:56:09,084-Speed 1677.31 samples/sec Loss 4.3642 Epoch: 14 Global Step: 69750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:56:36,128-Speed 1893.29 samples/sec Loss 3.6421 Epoch: 14 Global Step: 69800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:57:02,628-Speed 1932.15 samples/sec Loss 3.6698 Epoch: 14 Global Step: 69850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:57:28,879-Speed 1950.42 samples/sec Loss 3.6515 Epoch: 14 Global Step: 69900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:57:55,398-Speed 1930.73 samples/sec Loss 3.7095 Epoch: 14 Global Step: 69950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:58:21,881-Speed 1933.40 samples/sec Loss 3.7086 Epoch: 14 Global Step: 70000 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 19:59:25,111-[lfw][70000]XNorm: 21.777602 Training: 2021-03-15 19:59:25,112-[lfw][70000]Accuracy-Flip: 0.99783+-0.00269 Training: 2021-03-15 19:59:25,112-[lfw][70000]Accuracy-Highest: 0.99833 Training: 2021-03-15 20:00:39,159-[cfp_fp][70000]XNorm: 19.811517 Training: 2021-03-15 20:00:39,159-[cfp_fp][70000]Accuracy-Flip: 0.98571+-0.00383 Training: 2021-03-15 20:00:39,159-[cfp_fp][70000]Accuracy-Highest: 0.98657 Training: 2021-03-15 20:01:43,003-[agedb_30][70000]XNorm: 21.728657 Training: 2021-03-15 20:01:43,004-[agedb_30][70000]Accuracy-Flip: 0.97983+-0.00689 Training: 2021-03-15 20:01:43,004-[agedb_30][70000]Accuracy-Highest: 0.98067 Training: 2021-03-15 20:02:08,871-Speed 225.56 samples/sec Loss 3.8360 Epoch: 14 Global Step: 70050 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:02:35,057-Speed 1955.29 samples/sec Loss 3.7671 Epoch: 14 Global Step: 70100 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:03:01,171-Speed 1960.74 samples/sec Loss 3.8359 Epoch: 14 Global Step: 70150 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:03:27,185-Speed 1968.24 samples/sec Loss 3.8519 Epoch: 14 Global Step: 70200 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:03:53,916-Speed 1915.44 samples/sec Loss 3.8925 Epoch: 14 Global Step: 70250 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:04:20,342-Speed 1937.51 samples/sec Loss 3.9440 Epoch: 14 Global Step: 70300 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:04:46,398-Speed 1965.05 samples/sec Loss 3.9322 Epoch: 14 Global Step: 70350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:05:13,149-Speed 1914.05 samples/sec Loss 3.9852 Epoch: 14 Global Step: 70400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:05:39,638-Speed 1932.93 samples/sec Loss 3.9091 Epoch: 14 Global Step: 70450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:06:05,837-Speed 1954.32 samples/sec Loss 3.9732 Epoch: 14 Global Step: 70500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:06:32,489-Speed 1921.11 samples/sec Loss 3.9925 Epoch: 14 Global Step: 70550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:06:58,492-Speed 1969.01 samples/sec Loss 4.0137 Epoch: 14 Global Step: 70600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:07:24,963-Speed 1934.26 samples/sec Loss 4.0109 Epoch: 14 Global Step: 70650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:07:51,072-Speed 1961.11 samples/sec Loss 4.0543 Epoch: 14 Global Step: 70700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:08:17,219-Speed 1958.32 samples/sec Loss 4.0204 Epoch: 14 Global Step: 70750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:08:43,243-Speed 1967.48 samples/sec Loss 4.0786 Epoch: 14 Global Step: 70800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:09:09,585-Speed 1943.75 samples/sec Loss 4.1007 Epoch: 14 Global Step: 70850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:09:36,160-Speed 1926.67 samples/sec Loss 4.0874 Epoch: 14 Global Step: 70900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:10:02,329-Speed 1956.56 samples/sec Loss 4.1160 Epoch: 14 Global Step: 70950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:10:29,056-Speed 1915.76 samples/sec Loss 4.1143 Epoch: 14 Global Step: 71000 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:10:55,537-Speed 1933.47 samples/sec Loss 4.1651 Epoch: 14 Global Step: 71050 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:11:21,826-Speed 1947.69 samples/sec Loss 4.1220 Epoch: 14 Global Step: 71100 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:11:48,297-Speed 1934.26 samples/sec Loss 4.1464 Epoch: 14 Global Step: 71150 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:12:14,734-Speed 1936.71 samples/sec Loss 4.1583 Epoch: 14 Global Step: 71200 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:12:41,629-Speed 1903.81 samples/sec Loss 4.1729 Epoch: 14 Global Step: 71250 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:13:08,143-Speed 1931.08 samples/sec Loss 4.1729 Epoch: 14 Global Step: 71300 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:13:34,544-Speed 1939.39 samples/sec Loss 4.2024 Epoch: 14 Global Step: 71350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:14:01,315-Speed 1912.59 samples/sec Loss 4.1919 Epoch: 14 Global Step: 71400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:14:27,622-Speed 1946.28 samples/sec Loss 4.1746 Epoch: 14 Global Step: 71450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:14:54,241-Speed 1923.50 samples/sec Loss 4.2048 Epoch: 14 Global Step: 71500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:15:20,431-Speed 1955.03 samples/sec Loss 4.1741 Epoch: 14 Global Step: 71550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:15:46,700-Speed 1949.13 samples/sec Loss 4.1921 Epoch: 14 Global Step: 71600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:16:12,774-Speed 1963.68 samples/sec Loss 4.1524 Epoch: 14 Global Step: 71650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:16:38,960-Speed 1955.37 samples/sec Loss 4.1866 Epoch: 14 Global Step: 71700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:17:05,162-Speed 1954.26 samples/sec Loss 4.2752 Epoch: 14 Global Step: 71750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:17:31,382-Speed 1952.75 samples/sec Loss 4.2094 Epoch: 14 Global Step: 71800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:17:57,968-Speed 1925.85 samples/sec Loss 4.1919 Epoch: 14 Global Step: 71850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:18:24,336-Speed 1941.85 samples/sec Loss 4.2732 Epoch: 14 Global Step: 71900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:18:50,881-Speed 1928.85 samples/sec Loss 4.2577 Epoch: 14 Global Step: 71950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:19:16,922-Speed 1966.30 samples/sec Loss 4.1838 Epoch: 14 Global Step: 72000 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:20:21,914-[lfw][72000]XNorm: 22.369570 Training: 2021-03-15 20:20:21,914-[lfw][72000]Accuracy-Flip: 0.99767+-0.00281 Training: 2021-03-15 20:20:21,914-[lfw][72000]Accuracy-Highest: 0.99833 Training: 2021-03-15 20:21:35,158-[cfp_fp][72000]XNorm: 20.295119 Training: 2021-03-15 20:21:35,158-[cfp_fp][72000]Accuracy-Flip: 0.98714+-0.00306 Training: 2021-03-15 20:21:35,158-[cfp_fp][72000]Accuracy-Highest: 0.98714 Training: 2021-03-15 20:22:40,122-[agedb_30][72000]XNorm: 22.310980 Training: 2021-03-15 20:22:40,122-[agedb_30][72000]Accuracy-Flip: 0.97967+-0.00670 Training: 2021-03-15 20:22:40,122-[agedb_30][72000]Accuracy-Highest: 0.98067 Training: 2021-03-15 20:23:06,971-Speed 222.56 samples/sec Loss 4.2758 Epoch: 14 Global Step: 72050 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:23:33,255-Speed 1948.02 samples/sec Loss 4.2514 Epoch: 14 Global Step: 72100 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:23:59,767-Speed 1931.24 samples/sec Loss 4.2223 Epoch: 14 Global Step: 72150 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:24:26,052-Speed 1947.92 samples/sec Loss 4.2575 Epoch: 14 Global Step: 72200 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:24:52,317-Speed 1949.39 samples/sec Loss 4.2892 Epoch: 14 Global Step: 72250 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:25:18,440-Speed 1960.18 samples/sec Loss 4.2794 Epoch: 14 Global Step: 72300 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:25:44,598-Speed 1957.39 samples/sec Loss 4.2447 Epoch: 14 Global Step: 72350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:26:10,442-Speed 1981.19 samples/sec Loss 4.2752 Epoch: 14 Global Step: 72400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:26:36,909-Speed 1934.58 samples/sec Loss 4.2654 Epoch: 14 Global Step: 72450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:27:03,702-Speed 1911.00 samples/sec Loss 4.3184 Epoch: 14 Global Step: 72500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:27:29,777-Speed 1963.60 samples/sec Loss 4.2508 Epoch: 14 Global Step: 72550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:27:55,713-Speed 1974.17 samples/sec Loss 4.3133 Epoch: 14 Global Step: 72600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:28:21,783-Speed 1963.98 samples/sec Loss 4.3100 Epoch: 14 Global Step: 72650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:28:48,460-Speed 1919.33 samples/sec Loss 4.3117 Epoch: 14 Global Step: 72700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:29:14,575-Speed 1960.62 samples/sec Loss 4.2695 Epoch: 14 Global Step: 72750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:29:41,532-Speed 1899.36 samples/sec Loss 4.3076 Epoch: 14 Global Step: 72800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:30:08,017-Speed 1933.26 samples/sec Loss 4.3186 Epoch: 14 Global Step: 72850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:30:34,258-Speed 1951.26 samples/sec Loss 4.2800 Epoch: 14 Global Step: 72900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:31:00,802-Speed 1929.02 samples/sec Loss 4.2964 Epoch: 14 Global Step: 72950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:31:26,884-Speed 1963.11 samples/sec Loss 4.3710 Epoch: 14 Global Step: 73000 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:31:53,569-Speed 1918.72 samples/sec Loss 4.3038 Epoch: 14 Global Step: 73050 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:32:19,473-Speed 1976.59 samples/sec Loss 4.2980 Epoch: 14 Global Step: 73100 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:32:46,093-Speed 1923.44 samples/sec Loss 4.2748 Epoch: 14 Global Step: 73150 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:33:12,668-Speed 1926.74 samples/sec Loss 4.2878 Epoch: 14 Global Step: 73200 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:33:39,060-Speed 1940.06 samples/sec Loss 4.3123 Epoch: 14 Global Step: 73250 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:34:05,597-Speed 1929.41 samples/sec Loss 4.3174 Epoch: 14 Global Step: 73300 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:34:31,807-Speed 1953.54 samples/sec Loss 4.2971 Epoch: 14 Global Step: 73350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:34:57,983-Speed 1956.21 samples/sec Loss 4.3422 Epoch: 14 Global Step: 73400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:35:24,510-Speed 1930.14 samples/sec Loss 4.3075 Epoch: 14 Global Step: 73450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:35:51,031-Speed 1930.76 samples/sec Loss 4.3157 Epoch: 14 Global Step: 73500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:36:17,289-Speed 1949.92 samples/sec Loss 4.3351 Epoch: 14 Global Step: 73550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:36:43,556-Speed 1949.30 samples/sec Loss 4.3363 Epoch: 14 Global Step: 73600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:37:10,182-Speed 1923.06 samples/sec Loss 4.3030 Epoch: 14 Global Step: 73650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:37:36,587-Speed 1939.14 samples/sec Loss 4.3237 Epoch: 14 Global Step: 73700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:38:03,062-Speed 1933.95 samples/sec Loss 4.3096 Epoch: 14 Global Step: 73750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:38:29,524-Speed 1934.98 samples/sec Loss 4.3004 Epoch: 14 Global Step: 73800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:38:56,035-Speed 1931.34 samples/sec Loss 4.2975 Epoch: 14 Global Step: 73850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:39:22,295-Speed 1949.80 samples/sec Loss 4.3431 Epoch: 14 Global Step: 73900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:39:48,632-Speed 1944.10 samples/sec Loss 4.2820 Epoch: 14 Global Step: 73950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:40:14,950-Speed 1945.47 samples/sec Loss 4.3405 Epoch: 14 Global Step: 74000 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:41:17,997-[lfw][74000]XNorm: 22.330151 Training: 2021-03-15 20:41:17,998-[lfw][74000]Accuracy-Flip: 0.99667+-0.00342 Training: 2021-03-15 20:41:17,998-[lfw][74000]Accuracy-Highest: 0.99833 Training: 2021-03-15 20:42:29,790-[cfp_fp][74000]XNorm: 20.012682 Training: 2021-03-15 20:42:29,790-[cfp_fp][74000]Accuracy-Flip: 0.98714+-0.00300 Training: 2021-03-15 20:42:29,790-[cfp_fp][74000]Accuracy-Highest: 0.98714 Training: 2021-03-15 20:43:31,918-[agedb_30][74000]XNorm: 22.350554 Training: 2021-03-15 20:43:31,919-[agedb_30][74000]Accuracy-Flip: 0.97983+-0.00717 Training: 2021-03-15 20:43:31,929-[agedb_30][74000]Accuracy-Highest: 0.98067 Training: 2021-03-15 20:43:58,535-Speed 229.00 samples/sec Loss 4.3366 Epoch: 14 Global Step: 74050 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:44:24,397-Speed 1979.82 samples/sec Loss 4.3006 Epoch: 14 Global Step: 74100 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:44:50,905-Speed 1931.55 samples/sec Loss 4.3588 Epoch: 14 Global Step: 74150 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:45:17,397-Speed 1932.69 samples/sec Loss 4.3234 Epoch: 14 Global Step: 74200 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:45:43,745-Speed 1943.29 samples/sec Loss 4.3489 Epoch: 14 Global Step: 74250 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:46:09,996-Speed 1950.45 samples/sec Loss 4.2801 Epoch: 14 Global Step: 74300 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:46:36,190-Speed 1954.74 samples/sec Loss 4.3972 Epoch: 14 Global Step: 74350 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:47:02,447-Speed 1950.00 samples/sec Loss 4.3035 Epoch: 14 Global Step: 74400 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:47:28,905-Speed 1935.26 samples/sec Loss 4.3746 Epoch: 14 Global Step: 74450 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:47:55,191-Speed 1947.84 samples/sec Loss 4.2718 Epoch: 14 Global Step: 74500 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:48:21,206-Speed 1968.18 samples/sec Loss 4.2832 Epoch: 14 Global Step: 74550 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:48:47,358-Speed 1957.85 samples/sec Loss 4.2725 Epoch: 14 Global Step: 74600 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:49:13,670-Speed 1945.90 samples/sec Loss 4.3257 Epoch: 14 Global Step: 74650 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:49:39,814-Speed 1958.51 samples/sec Loss 4.3018 Epoch: 14 Global Step: 74700 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:50:10,927-Speed 1645.64 samples/sec Loss 4.0667 Epoch: 15 Global Step: 74750 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:50:36,981-Speed 1965.26 samples/sec Loss 3.5823 Epoch: 15 Global Step: 74800 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:51:03,629-Speed 1921.42 samples/sec Loss 3.6099 Epoch: 15 Global Step: 74850 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:51:29,659-Speed 1967.00 samples/sec Loss 3.6758 Epoch: 15 Global Step: 74900 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:51:55,976-Speed 1945.53 samples/sec Loss 3.6814 Epoch: 15 Global Step: 74950 Fp16 Grad Scale: 16384 Required: 8 hours Training: 2021-03-15 20:52:22,404-Speed 1937.51 samples/sec Loss 3.7179 Epoch: 15 Global Step: 75000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:52:48,614-Speed 1953.53 samples/sec Loss 3.7190 Epoch: 15 Global Step: 75050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:53:14,845-Speed 1951.94 samples/sec Loss 3.7731 Epoch: 15 Global Step: 75100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:53:41,178-Speed 1944.42 samples/sec Loss 3.8184 Epoch: 15 Global Step: 75150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:54:07,423-Speed 1950.99 samples/sec Loss 3.8194 Epoch: 15 Global Step: 75200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:54:34,112-Speed 1918.46 samples/sec Loss 3.8076 Epoch: 15 Global Step: 75250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:55:00,496-Speed 1940.75 samples/sec Loss 3.8793 Epoch: 15 Global Step: 75300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:55:26,622-Speed 1959.84 samples/sec Loss 3.8997 Epoch: 15 Global Step: 75350 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:55:53,222-Speed 1924.88 samples/sec Loss 3.8975 Epoch: 15 Global Step: 75400 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:56:19,286-Speed 1964.45 samples/sec Loss 3.8807 Epoch: 15 Global Step: 75450 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:56:45,399-Speed 1960.75 samples/sec Loss 3.9497 Epoch: 15 Global Step: 75500 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:57:11,698-Speed 1946.94 samples/sec Loss 3.9492 Epoch: 15 Global Step: 75550 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:57:38,064-Speed 1941.93 samples/sec Loss 3.9721 Epoch: 15 Global Step: 75600 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:58:04,366-Speed 1946.68 samples/sec Loss 4.0056 Epoch: 15 Global Step: 75650 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:58:30,718-Speed 1942.96 samples/sec Loss 4.0294 Epoch: 15 Global Step: 75700 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:58:57,021-Speed 1946.60 samples/sec Loss 4.0127 Epoch: 15 Global Step: 75750 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:59:23,613-Speed 1925.54 samples/sec Loss 4.0382 Epoch: 15 Global Step: 75800 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 20:59:49,909-Speed 1947.14 samples/sec Loss 4.0368 Epoch: 15 Global Step: 75850 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:00:16,064-Speed 1957.60 samples/sec Loss 4.0310 Epoch: 15 Global Step: 75900 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:00:42,410-Speed 1943.44 samples/sec Loss 4.0824 Epoch: 15 Global Step: 75950 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:01:08,739-Speed 1944.68 samples/sec Loss 4.0460 Epoch: 15 Global Step: 76000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:02:13,948-[lfw][76000]XNorm: 20.902923 Training: 2021-03-15 21:02:13,949-[lfw][76000]Accuracy-Flip: 0.99800+-0.00256 Training: 2021-03-15 21:02:13,949-[lfw][76000]Accuracy-Highest: 0.99833 Training: 2021-03-15 21:03:29,160-[cfp_fp][76000]XNorm: 19.348317 Training: 2021-03-15 21:03:29,160-[cfp_fp][76000]Accuracy-Flip: 0.98757+-0.00389 Training: 2021-03-15 21:03:29,160-[cfp_fp][76000]Accuracy-Highest: 0.98757 Training: 2021-03-15 21:04:32,945-[agedb_30][76000]XNorm: 21.259406 Training: 2021-03-15 21:04:32,945-[agedb_30][76000]Accuracy-Flip: 0.98267+-0.00768 Training: 2021-03-15 21:04:32,945-[agedb_30][76000]Accuracy-Highest: 0.98267 Training: 2021-03-15 21:04:59,056-Speed 222.30 samples/sec Loss 4.0637 Epoch: 15 Global Step: 76050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:05:25,027-Speed 1971.49 samples/sec Loss 4.0784 Epoch: 15 Global Step: 76100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:05:51,498-Speed 1934.23 samples/sec Loss 4.1197 Epoch: 15 Global Step: 76150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:06:17,802-Speed 1946.54 samples/sec Loss 4.0512 Epoch: 15 Global Step: 76200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:06:44,094-Speed 1947.54 samples/sec Loss 4.1249 Epoch: 15 Global Step: 76250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:07:10,622-Speed 1930.22 samples/sec Loss 4.1424 Epoch: 15 Global Step: 76300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:07:36,951-Speed 1944.72 samples/sec Loss 4.1033 Epoch: 15 Global Step: 76350 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:08:03,232-Speed 1948.27 samples/sec Loss 4.0926 Epoch: 15 Global Step: 76400 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:08:29,664-Speed 1937.10 samples/sec Loss 4.1648 Epoch: 15 Global Step: 76450 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:08:56,210-Speed 1928.81 samples/sec Loss 4.1370 Epoch: 15 Global Step: 76500 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:09:22,385-Speed 1956.12 samples/sec Loss 4.1252 Epoch: 15 Global Step: 76550 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:09:48,685-Speed 1946.80 samples/sec Loss 4.1788 Epoch: 15 Global Step: 76600 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:10:14,943-Speed 1949.94 samples/sec Loss 4.1123 Epoch: 15 Global Step: 76650 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:10:41,177-Speed 1951.82 samples/sec Loss 4.1683 Epoch: 15 Global Step: 76700 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:11:07,408-Speed 1951.95 samples/sec Loss 4.1909 Epoch: 15 Global Step: 76750 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:11:33,524-Speed 1960.51 samples/sec Loss 4.2229 Epoch: 15 Global Step: 76800 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:12:00,272-Speed 1914.26 samples/sec Loss 4.1551 Epoch: 15 Global Step: 76850 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:12:26,492-Speed 1952.76 samples/sec Loss 4.2246 Epoch: 15 Global Step: 76900 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:12:52,952-Speed 1935.14 samples/sec Loss 4.1977 Epoch: 15 Global Step: 76950 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:13:19,213-Speed 1949.68 samples/sec Loss 4.1823 Epoch: 15 Global Step: 77000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:13:45,735-Speed 1930.56 samples/sec Loss 4.2047 Epoch: 15 Global Step: 77050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:14:11,990-Speed 1950.17 samples/sec Loss 4.2117 Epoch: 15 Global Step: 77100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:14:38,334-Speed 1943.60 samples/sec Loss 4.2296 Epoch: 15 Global Step: 77150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:15:04,459-Speed 1959.88 samples/sec Loss 4.1845 Epoch: 15 Global Step: 77200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:15:30,851-Speed 1939.98 samples/sec Loss 4.1898 Epoch: 15 Global Step: 77250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:15:57,636-Speed 1911.63 samples/sec Loss 4.2199 Epoch: 15 Global Step: 77300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:16:23,999-Speed 1943.25 samples/sec Loss 4.2161 Epoch: 15 Global Step: 77350 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:16:50,460-Speed 1934.95 samples/sec Loss 4.2443 Epoch: 15 Global Step: 77400 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:17:17,590-Speed 1887.30 samples/sec Loss 4.2051 Epoch: 15 Global Step: 77450 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:17:43,776-Speed 1955.26 samples/sec Loss 4.2519 Epoch: 15 Global Step: 77500 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:18:10,366-Speed 1925.66 samples/sec Loss 4.2479 Epoch: 15 Global Step: 77550 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:18:36,474-Speed 1961.12 samples/sec Loss 4.2207 Epoch: 15 Global Step: 77600 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:19:02,568-Speed 1962.39 samples/sec Loss 4.2289 Epoch: 15 Global Step: 77650 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:19:29,056-Speed 1933.00 samples/sec Loss 4.1972 Epoch: 15 Global Step: 77700 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:19:54,957-Speed 1976.77 samples/sec Loss 4.2770 Epoch: 15 Global Step: 77750 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:20:21,662-Speed 1917.32 samples/sec Loss 4.3043 Epoch: 15 Global Step: 77800 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:20:48,002-Speed 1943.83 samples/sec Loss 4.2344 Epoch: 15 Global Step: 77850 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:21:13,746-Speed 1989.00 samples/sec Loss 4.2549 Epoch: 15 Global Step: 77900 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:21:40,154-Speed 1938.84 samples/sec Loss 4.2425 Epoch: 15 Global Step: 77950 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:22:06,490-Speed 1944.35 samples/sec Loss 4.2681 Epoch: 15 Global Step: 78000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:23:09,987-[lfw][78000]XNorm: 22.082125 Training: 2021-03-15 21:23:09,988-[lfw][78000]Accuracy-Flip: 0.99817+-0.00229 Training: 2021-03-15 21:23:09,988-[lfw][78000]Accuracy-Highest: 0.99833 Training: 2021-03-15 21:24:24,146-[cfp_fp][78000]XNorm: 20.130121 Training: 2021-03-15 21:24:24,146-[cfp_fp][78000]Accuracy-Flip: 0.98429+-0.00409 Training: 2021-03-15 21:24:24,146-[cfp_fp][78000]Accuracy-Highest: 0.98757 Training: 2021-03-15 21:25:27,765-[agedb_30][78000]XNorm: 22.107413 Training: 2021-03-15 21:25:27,766-[agedb_30][78000]Accuracy-Flip: 0.98050+-0.00650 Training: 2021-03-15 21:25:27,766-[agedb_30][78000]Accuracy-Highest: 0.98267 Training: 2021-03-15 21:25:54,357-Speed 224.69 samples/sec Loss 4.2675 Epoch: 15 Global Step: 78050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:26:20,668-Speed 1946.03 samples/sec Loss 4.2936 Epoch: 15 Global Step: 78100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:26:46,742-Speed 1963.75 samples/sec Loss 4.2587 Epoch: 15 Global Step: 78150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:27:13,216-Speed 1934.07 samples/sec Loss 4.3216 Epoch: 15 Global Step: 78200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:27:39,429-Speed 1953.29 samples/sec Loss 4.2469 Epoch: 15 Global Step: 78250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:28:05,845-Speed 1938.31 samples/sec Loss 4.2771 Epoch: 15 Global Step: 78300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:28:32,094-Speed 1950.63 samples/sec Loss 4.2677 Epoch: 15 Global Step: 78350 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:28:58,354-Speed 1949.74 samples/sec Loss 4.2527 Epoch: 15 Global Step: 78400 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:29:24,626-Speed 1948.90 samples/sec Loss 4.2639 Epoch: 15 Global Step: 78450 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:29:51,220-Speed 1925.33 samples/sec Loss 4.2724 Epoch: 15 Global Step: 78500 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:30:17,633-Speed 1938.48 samples/sec Loss 4.2808 Epoch: 15 Global Step: 78550 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:30:43,950-Speed 1945.57 samples/sec Loss 4.2863 Epoch: 15 Global Step: 78600 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:31:10,553-Speed 1924.71 samples/sec Loss 4.2640 Epoch: 15 Global Step: 78650 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:31:36,856-Speed 1946.57 samples/sec Loss 4.2841 Epoch: 15 Global Step: 78700 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:32:02,996-Speed 1958.76 samples/sec Loss 4.2782 Epoch: 15 Global Step: 78750 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:32:29,404-Speed 1938.93 samples/sec Loss 4.2341 Epoch: 15 Global Step: 78800 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:32:55,648-Speed 1950.96 samples/sec Loss 4.2739 Epoch: 15 Global Step: 78850 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:33:22,123-Speed 1933.93 samples/sec Loss 4.2918 Epoch: 15 Global Step: 78900 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:33:48,678-Speed 1928.13 samples/sec Loss 4.2640 Epoch: 15 Global Step: 78950 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:34:14,419-Speed 1989.12 samples/sec Loss 4.3089 Epoch: 15 Global Step: 79000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:34:40,638-Speed 1952.85 samples/sec Loss 4.2875 Epoch: 15 Global Step: 79050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:35:07,417-Speed 1911.98 samples/sec Loss 4.2514 Epoch: 15 Global Step: 79100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:35:33,587-Speed 1956.55 samples/sec Loss 4.2662 Epoch: 15 Global Step: 79150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:35:59,941-Speed 1942.80 samples/sec Loss 4.2455 Epoch: 15 Global Step: 79200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:36:26,382-Speed 1936.48 samples/sec Loss 4.2749 Epoch: 15 Global Step: 79250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:36:52,341-Speed 1972.40 samples/sec Loss 4.2463 Epoch: 15 Global Step: 79300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:37:18,425-Speed 1962.93 samples/sec Loss 4.3285 Epoch: 15 Global Step: 79350 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:37:44,645-Speed 1952.79 samples/sec Loss 4.2655 Epoch: 15 Global Step: 79400 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:38:10,984-Speed 1943.93 samples/sec Loss 4.2747 Epoch: 15 Global Step: 79450 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:38:37,664-Speed 1919.11 samples/sec Loss 4.2952 Epoch: 15 Global Step: 79500 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:39:04,129-Speed 1934.70 samples/sec Loss 4.2885 Epoch: 15 Global Step: 79550 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:39:30,444-Speed 1945.71 samples/sec Loss 4.2788 Epoch: 15 Global Step: 79600 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:39:56,595-Speed 1957.95 samples/sec Loss 4.2773 Epoch: 15 Global Step: 79650 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:40:22,884-Speed 1947.61 samples/sec Loss 4.2398 Epoch: 15 Global Step: 79700 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:40:54,081-Speed 1641.23 samples/sec Loss 3.5399 Epoch: 16 Global Step: 79750 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:41:19,997-Speed 1975.68 samples/sec Loss 3.0126 Epoch: 16 Global Step: 79800 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:41:46,748-Speed 1914.03 samples/sec Loss 2.9743 Epoch: 16 Global Step: 79850 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:42:13,507-Speed 1913.45 samples/sec Loss 2.8278 Epoch: 16 Global Step: 79900 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:42:39,646-Speed 1958.82 samples/sec Loss 2.8186 Epoch: 16 Global Step: 79950 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:43:05,886-Speed 1951.27 samples/sec Loss 2.7674 Epoch: 16 Global Step: 80000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:44:09,559-[lfw][80000]XNorm: 22.572459 Training: 2021-03-15 21:44:09,560-[lfw][80000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-15 21:44:09,560-[lfw][80000]Accuracy-Highest: 0.99833 Training: 2021-03-15 21:45:24,729-[cfp_fp][80000]XNorm: 20.822278 Training: 2021-03-15 21:45:24,729-[cfp_fp][80000]Accuracy-Flip: 0.98657+-0.00232 Training: 2021-03-15 21:45:24,729-[cfp_fp][80000]Accuracy-Highest: 0.98757 Training: 2021-03-15 21:46:28,662-[agedb_30][80000]XNorm: 22.677464 Training: 2021-03-15 21:46:28,663-[agedb_30][80000]Accuracy-Flip: 0.98183+-0.00669 Training: 2021-03-15 21:46:28,663-[agedb_30][80000]Accuracy-Highest: 0.98267 Training: 2021-03-15 21:46:54,944-Speed 223.53 samples/sec Loss 2.7509 Epoch: 16 Global Step: 80050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:47:20,977-Speed 1966.77 samples/sec Loss 2.7378 Epoch: 16 Global Step: 80100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:47:46,967-Speed 1970.04 samples/sec Loss 2.7104 Epoch: 16 Global Step: 80150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:48:12,998-Speed 1966.96 samples/sec Loss 2.6819 Epoch: 16 Global Step: 80200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:48:39,090-Speed 1962.33 samples/sec Loss 2.6248 Epoch: 16 Global Step: 80250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:49:05,309-Speed 1952.87 samples/sec Loss 2.6561 Epoch: 16 Global Step: 80300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:49:31,688-Speed 1941.00 samples/sec Loss 2.6395 Epoch: 16 Global Step: 80350 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:49:58,158-Speed 1934.33 samples/sec Loss 2.5598 Epoch: 16 Global Step: 80400 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:50:24,233-Speed 1963.72 samples/sec Loss 2.6062 Epoch: 16 Global Step: 80450 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:50:50,666-Speed 1937.03 samples/sec Loss 2.5485 Epoch: 16 Global Step: 80500 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:51:17,015-Speed 1943.19 samples/sec Loss 2.6111 Epoch: 16 Global Step: 80550 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:51:43,484-Speed 1934.40 samples/sec Loss 2.5241 Epoch: 16 Global Step: 80600 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:52:09,627-Speed 1958.57 samples/sec Loss 2.5674 Epoch: 16 Global Step: 80650 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:52:36,779-Speed 1885.68 samples/sec Loss 2.5167 Epoch: 16 Global Step: 80700 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:53:03,048-Speed 1949.16 samples/sec Loss 2.4601 Epoch: 16 Global Step: 80750 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:53:29,389-Speed 1943.78 samples/sec Loss 2.4605 Epoch: 16 Global Step: 80800 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:53:55,757-Speed 1941.85 samples/sec Loss 2.5047 Epoch: 16 Global Step: 80850 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:54:22,003-Speed 1950.80 samples/sec Loss 2.4508 Epoch: 16 Global Step: 80900 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:54:48,136-Speed 1959.26 samples/sec Loss 2.4471 Epoch: 16 Global Step: 80950 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:55:14,575-Speed 1936.61 samples/sec Loss 2.4116 Epoch: 16 Global Step: 81000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:55:40,988-Speed 1938.50 samples/sec Loss 2.4106 Epoch: 16 Global Step: 81050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:56:07,589-Speed 1924.76 samples/sec Loss 2.4328 Epoch: 16 Global Step: 81100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:56:34,016-Speed 1937.52 samples/sec Loss 2.4218 Epoch: 16 Global Step: 81150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:57:00,468-Speed 1935.71 samples/sec Loss 2.3967 Epoch: 16 Global Step: 81200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:57:26,659-Speed 1954.90 samples/sec Loss 2.4419 Epoch: 16 Global Step: 81250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:57:53,029-Speed 1941.70 samples/sec Loss 2.3709 Epoch: 16 Global Step: 81300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:58:19,537-Speed 1931.51 samples/sec Loss 2.3237 Epoch: 16 Global Step: 81350 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:58:45,986-Speed 1935.86 samples/sec Loss 2.3592 Epoch: 16 Global Step: 81400 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:59:12,248-Speed 1949.70 samples/sec Loss 2.3861 Epoch: 16 Global Step: 81450 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 21:59:38,821-Speed 1926.93 samples/sec Loss 2.3480 Epoch: 16 Global Step: 81500 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:00:05,174-Speed 1942.95 samples/sec Loss 2.3513 Epoch: 16 Global Step: 81550 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:00:31,580-Speed 1939.02 samples/sec Loss 2.3681 Epoch: 16 Global Step: 81600 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:00:57,988-Speed 1938.81 samples/sec Loss 2.3351 Epoch: 16 Global Step: 81650 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:01:24,205-Speed 1953.11 samples/sec Loss 2.3294 Epoch: 16 Global Step: 81700 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:01:50,082-Speed 1978.71 samples/sec Loss 2.3589 Epoch: 16 Global Step: 81750 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:02:16,958-Speed 1905.06 samples/sec Loss 2.3426 Epoch: 16 Global Step: 81800 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:02:43,001-Speed 1966.09 samples/sec Loss 2.3351 Epoch: 16 Global Step: 81850 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:03:09,039-Speed 1966.41 samples/sec Loss 2.3173 Epoch: 16 Global Step: 81900 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:03:35,309-Speed 1949.03 samples/sec Loss 2.2868 Epoch: 16 Global Step: 81950 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:04:01,860-Speed 1928.42 samples/sec Loss 2.2872 Epoch: 16 Global Step: 82000 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:05:06,391-[lfw][82000]XNorm: 22.316519 Training: 2021-03-15 22:05:06,392-[lfw][82000]Accuracy-Flip: 0.99817+-0.00241 Training: 2021-03-15 22:05:06,392-[lfw][82000]Accuracy-Highest: 0.99833 Training: 2021-03-15 22:06:21,020-[cfp_fp][82000]XNorm: 20.853778 Training: 2021-03-15 22:06:21,021-[cfp_fp][82000]Accuracy-Flip: 0.98943+-0.00294 Training: 2021-03-15 22:06:21,021-[cfp_fp][82000]Accuracy-Highest: 0.98943 Training: 2021-03-15 22:07:41,220-[agedb_30][82000]XNorm: 22.539650 Training: 2021-03-15 22:07:41,220-[agedb_30][82000]Accuracy-Flip: 0.98233+-0.00638 Training: 2021-03-15 22:07:41,220-[agedb_30][82000]Accuracy-Highest: 0.98267 Training: 2021-03-15 22:09:01,363-Speed 170.95 samples/sec Loss 2.3042 Epoch: 16 Global Step: 82050 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:09:34,774-Speed 1532.45 samples/sec Loss 2.3183 Epoch: 16 Global Step: 82100 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:10:01,191-Speed 1938.24 samples/sec Loss 2.2734 Epoch: 16 Global Step: 82150 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:10:27,589-Speed 1939.63 samples/sec Loss 2.2672 Epoch: 16 Global Step: 82200 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:10:53,898-Speed 1946.15 samples/sec Loss 2.3037 Epoch: 16 Global Step: 82250 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:11:20,865-Speed 1898.68 samples/sec Loss 2.2474 Epoch: 16 Global Step: 82300 Fp16 Grad Scale: 16384 Required: 7 hours Training: 2021-03-15 22:11:47,569-Speed 1917.32 samples/sec Loss 2.2809 Epoch: 16 Global Step: 82350 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:12:13,995-Speed 1937.55 samples/sec Loss 2.2576 Epoch: 16 Global Step: 82400 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:12:40,135-Speed 1958.78 samples/sec Loss 2.2722 Epoch: 16 Global Step: 82450 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:13:08,263-Speed 1820.29 samples/sec Loss 2.2389 Epoch: 16 Global Step: 82500 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:13:34,409-Speed 1958.31 samples/sec Loss 2.2595 Epoch: 16 Global Step: 82550 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:14:00,686-Speed 1948.66 samples/sec Loss 2.2505 Epoch: 16 Global Step: 82600 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:14:27,472-Speed 1911.52 samples/sec Loss 2.2118 Epoch: 16 Global Step: 82650 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:14:53,433-Speed 1972.25 samples/sec Loss 2.2349 Epoch: 16 Global Step: 82700 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:15:20,111-Speed 1919.27 samples/sec Loss 2.2078 Epoch: 16 Global Step: 82750 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:15:46,344-Speed 1951.86 samples/sec Loss 2.2206 Epoch: 16 Global Step: 82800 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:16:12,415-Speed 1963.97 samples/sec Loss 2.2375 Epoch: 16 Global Step: 82850 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:16:38,871-Speed 1935.33 samples/sec Loss 2.2170 Epoch: 16 Global Step: 82900 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:17:05,282-Speed 1938.63 samples/sec Loss 2.1719 Epoch: 16 Global Step: 82950 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:17:31,356-Speed 1963.71 samples/sec Loss 2.1528 Epoch: 16 Global Step: 83000 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:17:57,925-Speed 1927.15 samples/sec Loss 2.2371 Epoch: 16 Global Step: 83050 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:18:24,047-Speed 1960.08 samples/sec Loss 2.2133 Epoch: 16 Global Step: 83100 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:18:50,384-Speed 1944.09 samples/sec Loss 2.1364 Epoch: 16 Global Step: 83150 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:19:16,901-Speed 1930.88 samples/sec Loss 2.1509 Epoch: 16 Global Step: 83200 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:19:43,353-Speed 1935.64 samples/sec Loss 2.2273 Epoch: 16 Global Step: 83250 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:20:09,816-Speed 1934.85 samples/sec Loss 2.1594 Epoch: 16 Global Step: 83300 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:20:35,605-Speed 1985.42 samples/sec Loss 2.1733 Epoch: 16 Global Step: 83350 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:21:02,139-Speed 1929.70 samples/sec Loss 2.1727 Epoch: 16 Global Step: 83400 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:21:28,485-Speed 1943.40 samples/sec Loss 2.1391 Epoch: 16 Global Step: 83450 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:21:54,929-Speed 1936.26 samples/sec Loss 2.1062 Epoch: 16 Global Step: 83500 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:22:21,324-Speed 1939.77 samples/sec Loss 2.1020 Epoch: 16 Global Step: 83550 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:22:47,415-Speed 1962.43 samples/sec Loss 2.1438 Epoch: 16 Global Step: 83600 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:23:13,631-Speed 1953.09 samples/sec Loss 2.1437 Epoch: 16 Global Step: 83650 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:23:40,222-Speed 1925.52 samples/sec Loss 2.1394 Epoch: 16 Global Step: 83700 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:24:06,503-Speed 1948.23 samples/sec Loss 2.1181 Epoch: 16 Global Step: 83750 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:24:32,984-Speed 1933.59 samples/sec Loss 2.1380 Epoch: 16 Global Step: 83800 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:24:59,715-Speed 1915.44 samples/sec Loss 2.1196 Epoch: 16 Global Step: 83850 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:25:26,140-Speed 1937.58 samples/sec Loss 2.1326 Epoch: 16 Global Step: 83900 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:25:52,606-Speed 1934.61 samples/sec Loss 2.1623 Epoch: 16 Global Step: 83950 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:26:19,005-Speed 1939.54 samples/sec Loss 2.1113 Epoch: 16 Global Step: 84000 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:27:21,270-[lfw][84000]XNorm: 22.647718 Training: 2021-03-15 22:27:21,270-[lfw][84000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-15 22:27:21,270-[lfw][84000]Accuracy-Highest: 0.99833 Training: 2021-03-15 22:28:32,226-[cfp_fp][84000]XNorm: 21.246976 Training: 2021-03-15 22:28:32,226-[cfp_fp][84000]Accuracy-Flip: 0.98943+-0.00301 Training: 2021-03-15 22:28:32,226-[cfp_fp][84000]Accuracy-Highest: 0.98943 Training: 2021-03-15 22:29:35,309-[agedb_30][84000]XNorm: 22.945333 Training: 2021-03-15 22:29:35,309-[agedb_30][84000]Accuracy-Flip: 0.98250+-0.00724 Training: 2021-03-15 22:29:35,309-[agedb_30][84000]Accuracy-Highest: 0.98267 Training: 2021-03-15 22:30:01,638-Speed 229.98 samples/sec Loss 2.1145 Epoch: 16 Global Step: 84050 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:30:27,580-Speed 1973.78 samples/sec Loss 2.1071 Epoch: 16 Global Step: 84100 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:30:54,181-Speed 1924.77 samples/sec Loss 2.1440 Epoch: 16 Global Step: 84150 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:31:20,616-Speed 1936.91 samples/sec Loss 2.0820 Epoch: 16 Global Step: 84200 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:31:46,523-Speed 1976.55 samples/sec Loss 2.0941 Epoch: 16 Global Step: 84250 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:32:12,805-Speed 1948.16 samples/sec Loss 2.1143 Epoch: 16 Global Step: 84300 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:32:39,002-Speed 1954.48 samples/sec Loss 2.0720 Epoch: 16 Global Step: 84350 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:33:05,256-Speed 1950.26 samples/sec Loss 2.0874 Epoch: 16 Global Step: 84400 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:33:31,680-Speed 1937.64 samples/sec Loss 2.0815 Epoch: 16 Global Step: 84450 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:33:57,883-Speed 1954.08 samples/sec Loss 2.0921 Epoch: 16 Global Step: 84500 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:34:24,038-Speed 1957.59 samples/sec Loss 2.0936 Epoch: 16 Global Step: 84550 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:34:50,320-Speed 1948.18 samples/sec Loss 2.0645 Epoch: 16 Global Step: 84600 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:35:16,871-Speed 1928.46 samples/sec Loss 2.0952 Epoch: 16 Global Step: 84650 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:35:48,854-Speed 1600.87 samples/sec Loss 2.0263 Epoch: 17 Global Step: 84700 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:36:15,154-Speed 1946.84 samples/sec Loss 1.7490 Epoch: 17 Global Step: 84750 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:36:41,531-Speed 1941.15 samples/sec Loss 1.6984 Epoch: 17 Global Step: 84800 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:37:07,897-Speed 1942.00 samples/sec Loss 1.7230 Epoch: 17 Global Step: 84850 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:37:33,880-Speed 1970.62 samples/sec Loss 1.6996 Epoch: 17 Global Step: 84900 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:38:00,330-Speed 1935.75 samples/sec Loss 1.7207 Epoch: 17 Global Step: 84950 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:38:26,568-Speed 1951.50 samples/sec Loss 1.7245 Epoch: 17 Global Step: 85000 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:38:53,200-Speed 1922.58 samples/sec Loss 1.7195 Epoch: 17 Global Step: 85050 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:39:19,228-Speed 1967.13 samples/sec Loss 1.7362 Epoch: 17 Global Step: 85100 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:39:45,427-Speed 1954.46 samples/sec Loss 1.7248 Epoch: 17 Global Step: 85150 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:40:11,645-Speed 1952.90 samples/sec Loss 1.7468 Epoch: 17 Global Step: 85200 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:40:38,236-Speed 1925.55 samples/sec Loss 1.7188 Epoch: 17 Global Step: 85250 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:41:04,682-Speed 1936.13 samples/sec Loss 1.7040 Epoch: 17 Global Step: 85300 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:41:30,932-Speed 1950.55 samples/sec Loss 1.7117 Epoch: 17 Global Step: 85350 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:41:57,223-Speed 1947.43 samples/sec Loss 1.7303 Epoch: 17 Global Step: 85400 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:42:23,577-Speed 1942.85 samples/sec Loss 1.7253 Epoch: 17 Global Step: 85450 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:42:49,994-Speed 1938.25 samples/sec Loss 1.7240 Epoch: 17 Global Step: 85500 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:43:16,645-Speed 1921.32 samples/sec Loss 1.7144 Epoch: 17 Global Step: 85550 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:43:43,144-Speed 1932.22 samples/sec Loss 1.7081 Epoch: 17 Global Step: 85600 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:44:09,472-Speed 1944.78 samples/sec Loss 1.7201 Epoch: 17 Global Step: 85650 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:44:36,012-Speed 1929.22 samples/sec Loss 1.7307 Epoch: 17 Global Step: 85700 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:45:02,313-Speed 1946.84 samples/sec Loss 1.7446 Epoch: 17 Global Step: 85750 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:45:28,739-Speed 1937.58 samples/sec Loss 1.7571 Epoch: 17 Global Step: 85800 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:45:55,189-Speed 1935.80 samples/sec Loss 1.7057 Epoch: 17 Global Step: 85850 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:46:21,371-Speed 1955.59 samples/sec Loss 1.7099 Epoch: 17 Global Step: 85900 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:46:48,042-Speed 1919.74 samples/sec Loss 1.7432 Epoch: 17 Global Step: 85950 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:47:14,427-Speed 1940.54 samples/sec Loss 1.7327 Epoch: 17 Global Step: 86000 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:48:17,632-[lfw][86000]XNorm: 22.726955 Training: 2021-03-15 22:48:17,632-[lfw][86000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-15 22:48:17,634-[lfw][86000]Accuracy-Highest: 0.99833 Training: 2021-03-15 22:49:30,128-[cfp_fp][86000]XNorm: 21.380196 Training: 2021-03-15 22:49:30,128-[cfp_fp][86000]Accuracy-Flip: 0.98929+-0.00258 Training: 2021-03-15 22:49:30,128-[cfp_fp][86000]Accuracy-Highest: 0.98943 Training: 2021-03-15 22:50:32,356-[agedb_30][86000]XNorm: 23.110428 Training: 2021-03-15 22:50:32,356-[agedb_30][86000]Accuracy-Flip: 0.98267+-0.00712 Training: 2021-03-15 22:50:32,356-[agedb_30][86000]Accuracy-Highest: 0.98267 Training: 2021-03-15 22:50:58,722-Speed 228.27 samples/sec Loss 1.7256 Epoch: 17 Global Step: 86050 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:51:24,691-Speed 1971.69 samples/sec Loss 1.7024 Epoch: 17 Global Step: 86100 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:51:50,855-Speed 1956.95 samples/sec Loss 1.7334 Epoch: 17 Global Step: 86150 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:52:17,002-Speed 1958.23 samples/sec Loss 1.6808 Epoch: 17 Global Step: 86200 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:52:43,391-Speed 1940.26 samples/sec Loss 1.7399 Epoch: 17 Global Step: 86250 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:53:10,287-Speed 1903.65 samples/sec Loss 1.7417 Epoch: 17 Global Step: 86300 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:53:37,044-Speed 1913.57 samples/sec Loss 1.7369 Epoch: 17 Global Step: 86350 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:54:03,261-Speed 1953.03 samples/sec Loss 1.7503 Epoch: 17 Global Step: 86400 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:54:29,977-Speed 1916.53 samples/sec Loss 1.7453 Epoch: 17 Global Step: 86450 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:54:56,538-Speed 1927.67 samples/sec Loss 1.7085 Epoch: 17 Global Step: 86500 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:55:22,631-Speed 1962.27 samples/sec Loss 1.7376 Epoch: 17 Global Step: 86550 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:55:49,387-Speed 1913.66 samples/sec Loss 1.7211 Epoch: 17 Global Step: 86600 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:56:16,023-Speed 1922.32 samples/sec Loss 1.7652 Epoch: 17 Global Step: 86650 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:56:41,894-Speed 1979.04 samples/sec Loss 1.7163 Epoch: 17 Global Step: 86700 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:57:07,913-Speed 1967.83 samples/sec Loss 1.7410 Epoch: 17 Global Step: 86750 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:57:34,346-Speed 1937.09 samples/sec Loss 1.7240 Epoch: 17 Global Step: 86800 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:58:01,071-Speed 1915.87 samples/sec Loss 1.7153 Epoch: 17 Global Step: 86850 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:58:27,423-Speed 1942.99 samples/sec Loss 1.7222 Epoch: 17 Global Step: 86900 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:58:54,034-Speed 1924.23 samples/sec Loss 1.7447 Epoch: 17 Global Step: 86950 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:59:20,620-Speed 1925.90 samples/sec Loss 1.7339 Epoch: 17 Global Step: 87000 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 22:59:47,005-Speed 1940.53 samples/sec Loss 1.7344 Epoch: 17 Global Step: 87050 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:00:13,426-Speed 1937.95 samples/sec Loss 1.7261 Epoch: 17 Global Step: 87100 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:00:40,084-Speed 1920.65 samples/sec Loss 1.7149 Epoch: 17 Global Step: 87150 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:01:06,549-Speed 1934.75 samples/sec Loss 1.7253 Epoch: 17 Global Step: 87200 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:01:32,686-Speed 1958.92 samples/sec Loss 1.7275 Epoch: 17 Global Step: 87250 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:01:58,932-Speed 1950.83 samples/sec Loss 1.7154 Epoch: 17 Global Step: 87300 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:02:25,475-Speed 1929.03 samples/sec Loss 1.7334 Epoch: 17 Global Step: 87350 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:02:51,963-Speed 1932.99 samples/sec Loss 1.7096 Epoch: 17 Global Step: 87400 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:03:17,923-Speed 1972.34 samples/sec Loss 1.7231 Epoch: 17 Global Step: 87450 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:03:44,039-Speed 1960.53 samples/sec Loss 1.7269 Epoch: 17 Global Step: 87500 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:04:10,186-Speed 1958.21 samples/sec Loss 1.7236 Epoch: 17 Global Step: 87550 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:04:37,119-Speed 1901.11 samples/sec Loss 1.7229 Epoch: 17 Global Step: 87600 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:05:03,235-Speed 1960.51 samples/sec Loss 1.7380 Epoch: 17 Global Step: 87650 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:05:29,540-Speed 1946.66 samples/sec Loss 1.7168 Epoch: 17 Global Step: 87700 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:05:55,841-Speed 1946.73 samples/sec Loss 1.7394 Epoch: 17 Global Step: 87750 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:06:21,772-Speed 1974.53 samples/sec Loss 1.7134 Epoch: 17 Global Step: 87800 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:06:47,984-Speed 1953.38 samples/sec Loss 1.7314 Epoch: 17 Global Step: 87850 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:07:14,187-Speed 1954.06 samples/sec Loss 1.7438 Epoch: 17 Global Step: 87900 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:07:40,859-Speed 1919.67 samples/sec Loss 1.7238 Epoch: 17 Global Step: 87950 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:08:07,676-Speed 1909.33 samples/sec Loss 1.7171 Epoch: 17 Global Step: 88000 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:09:10,478-[lfw][88000]XNorm: 22.676308 Training: 2021-03-15 23:09:10,478-[lfw][88000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-15 23:09:10,478-[lfw][88000]Accuracy-Highest: 0.99833 Training: 2021-03-15 23:10:21,442-[cfp_fp][88000]XNorm: 21.529857 Training: 2021-03-15 23:10:21,443-[cfp_fp][88000]Accuracy-Flip: 0.99000+-0.00202 Training: 2021-03-15 23:10:21,443-[cfp_fp][88000]Accuracy-Highest: 0.99000 Training: 2021-03-15 23:11:24,126-[agedb_30][88000]XNorm: 23.150889 Training: 2021-03-15 23:11:24,126-[agedb_30][88000]Accuracy-Flip: 0.98400+-0.00750 Training: 2021-03-15 23:11:24,126-[agedb_30][88000]Accuracy-Highest: 0.98400 Training: 2021-03-15 23:11:50,891-Speed 229.38 samples/sec Loss 1.6662 Epoch: 17 Global Step: 88050 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:12:17,533-Speed 1921.94 samples/sec Loss 1.7117 Epoch: 17 Global Step: 88100 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:12:43,751-Speed 1952.90 samples/sec Loss 1.7271 Epoch: 17 Global Step: 88150 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:13:10,945-Speed 1882.85 samples/sec Loss 1.7400 Epoch: 17 Global Step: 88200 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:13:37,743-Speed 1910.66 samples/sec Loss 1.7212 Epoch: 17 Global Step: 88250 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:14:04,362-Speed 1923.51 samples/sec Loss 1.7255 Epoch: 17 Global Step: 88300 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:14:30,935-Speed 1926.85 samples/sec Loss 1.7324 Epoch: 17 Global Step: 88350 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:14:57,356-Speed 1937.96 samples/sec Loss 1.7310 Epoch: 17 Global Step: 88400 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:15:23,682-Speed 1944.87 samples/sec Loss 1.7244 Epoch: 17 Global Step: 88450 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:15:49,741-Speed 1964.80 samples/sec Loss 1.7013 Epoch: 17 Global Step: 88500 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:16:16,303-Speed 1927.70 samples/sec Loss 1.7436 Epoch: 17 Global Step: 88550 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:16:42,567-Speed 1949.48 samples/sec Loss 1.7045 Epoch: 17 Global Step: 88600 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:17:09,195-Speed 1922.80 samples/sec Loss 1.7118 Epoch: 17 Global Step: 88650 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:17:35,406-Speed 1953.45 samples/sec Loss 1.6996 Epoch: 17 Global Step: 88700 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:18:01,766-Speed 1942.41 samples/sec Loss 1.7196 Epoch: 17 Global Step: 88750 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:18:28,264-Speed 1932.29 samples/sec Loss 1.6808 Epoch: 17 Global Step: 88800 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:18:54,473-Speed 1953.64 samples/sec Loss 1.7566 Epoch: 17 Global Step: 88850 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:19:20,818-Speed 1943.45 samples/sec Loss 1.7213 Epoch: 17 Global Step: 88900 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:19:47,205-Speed 1940.41 samples/sec Loss 1.7193 Epoch: 17 Global Step: 88950 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:20:13,389-Speed 1955.48 samples/sec Loss 1.7430 Epoch: 17 Global Step: 89000 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:20:40,024-Speed 1922.31 samples/sec Loss 1.7182 Epoch: 17 Global Step: 89050 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:21:06,363-Speed 1944.04 samples/sec Loss 1.7274 Epoch: 17 Global Step: 89100 Fp16 Grad Scale: 16384 Required: 6 hours Training: 2021-03-15 23:21:32,900-Speed 1929.41 samples/sec Loss 1.6812 Epoch: 17 Global Step: 89150 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:21:59,203-Speed 1946.61 samples/sec Loss 1.6894 Epoch: 17 Global Step: 89200 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:22:25,398-Speed 1954.59 samples/sec Loss 1.7040 Epoch: 17 Global Step: 89250 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:22:51,959-Speed 1927.70 samples/sec Loss 1.6941 Epoch: 17 Global Step: 89300 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:23:18,183-Speed 1952.49 samples/sec Loss 1.7158 Epoch: 17 Global Step: 89350 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:23:44,649-Speed 1934.66 samples/sec Loss 1.6991 Epoch: 17 Global Step: 89400 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:24:11,016-Speed 1941.82 samples/sec Loss 1.7092 Epoch: 17 Global Step: 89450 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:24:37,211-Speed 1954.67 samples/sec Loss 1.7026 Epoch: 17 Global Step: 89500 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:25:03,643-Speed 1937.07 samples/sec Loss 1.7180 Epoch: 17 Global Step: 89550 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:25:29,776-Speed 1959.34 samples/sec Loss 1.7000 Epoch: 17 Global Step: 89600 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:25:56,623-Speed 1907.17 samples/sec Loss 1.7039 Epoch: 17 Global Step: 89650 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:26:27,750-Speed 1644.88 samples/sec Loss 1.5557 Epoch: 18 Global Step: 89700 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:26:54,345-Speed 1925.29 samples/sec Loss 1.3583 Epoch: 18 Global Step: 89750 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:27:20,653-Speed 1946.22 samples/sec Loss 1.3894 Epoch: 18 Global Step: 89800 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:27:47,194-Speed 1929.18 samples/sec Loss 1.3726 Epoch: 18 Global Step: 89850 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:28:13,287-Speed 1962.23 samples/sec Loss 1.3913 Epoch: 18 Global Step: 89900 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:28:39,778-Speed 1932.94 samples/sec Loss 1.4006 Epoch: 18 Global Step: 89950 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:29:06,225-Speed 1936.02 samples/sec Loss 1.3864 Epoch: 18 Global Step: 90000 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:30:09,227-[lfw][90000]XNorm: 22.491623 Training: 2021-03-15 23:30:09,228-[lfw][90000]Accuracy-Flip: 0.99817+-0.00241 Training: 2021-03-15 23:30:09,228-[lfw][90000]Accuracy-Highest: 0.99833 Training: 2021-03-15 23:31:21,726-[cfp_fp][90000]XNorm: 21.458228 Training: 2021-03-15 23:31:21,727-[cfp_fp][90000]Accuracy-Flip: 0.98971+-0.00284 Training: 2021-03-15 23:31:21,727-[cfp_fp][90000]Accuracy-Highest: 0.99000 Training: 2021-03-15 23:32:23,519-[agedb_30][90000]XNorm: 22.916591 Training: 2021-03-15 23:32:23,519-[agedb_30][90000]Accuracy-Flip: 0.98350+-0.00673 Training: 2021-03-15 23:32:23,519-[agedb_30][90000]Accuracy-Highest: 0.98400 Training: 2021-03-15 23:32:49,641-Speed 229.17 samples/sec Loss 1.4251 Epoch: 18 Global Step: 90050 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:33:15,862-Speed 1952.63 samples/sec Loss 1.3985 Epoch: 18 Global Step: 90100 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:33:42,212-Speed 1943.17 samples/sec Loss 1.3895 Epoch: 18 Global Step: 90150 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:34:08,530-Speed 1945.54 samples/sec Loss 1.4106 Epoch: 18 Global Step: 90200 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:34:34,902-Speed 1941.56 samples/sec Loss 1.4119 Epoch: 18 Global Step: 90250 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:35:01,204-Speed 1946.64 samples/sec Loss 1.4160 Epoch: 18 Global Step: 90300 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:35:27,472-Speed 1949.21 samples/sec Loss 1.4081 Epoch: 18 Global Step: 90350 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:35:54,322-Speed 1906.98 samples/sec Loss 1.3906 Epoch: 18 Global Step: 90400 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:36:20,530-Speed 1953.67 samples/sec Loss 1.4667 Epoch: 18 Global Step: 90450 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:36:46,848-Speed 1945.46 samples/sec Loss 1.4219 Epoch: 18 Global Step: 90500 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:37:13,262-Speed 1938.48 samples/sec Loss 1.4367 Epoch: 18 Global Step: 90550 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:37:39,825-Speed 1927.52 samples/sec Loss 1.4121 Epoch: 18 Global Step: 90600 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:38:06,214-Speed 1940.26 samples/sec Loss 1.4155 Epoch: 18 Global Step: 90650 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:38:32,513-Speed 1946.92 samples/sec Loss 1.4155 Epoch: 18 Global Step: 90700 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:39:09,346-Speed 1390.08 samples/sec Loss 1.4166 Epoch: 18 Global Step: 90750 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:39:35,503-Speed 1957.52 samples/sec Loss 1.4337 Epoch: 18 Global Step: 90800 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:40:01,827-Speed 1945.02 samples/sec Loss 1.4339 Epoch: 18 Global Step: 90850 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:40:28,324-Speed 1932.38 samples/sec Loss 1.4414 Epoch: 18 Global Step: 90900 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:40:55,094-Speed 1912.66 samples/sec Loss 1.4426 Epoch: 18 Global Step: 90950 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:41:21,355-Speed 1949.70 samples/sec Loss 1.4326 Epoch: 18 Global Step: 91000 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:41:47,904-Speed 1928.53 samples/sec Loss 1.4419 Epoch: 18 Global Step: 91050 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:42:14,053-Speed 1958.12 samples/sec Loss 1.4555 Epoch: 18 Global Step: 91100 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:42:40,191-Speed 1959.08 samples/sec Loss 1.4402 Epoch: 18 Global Step: 91150 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:43:06,384-Speed 1954.76 samples/sec Loss 1.4604 Epoch: 18 Global Step: 91200 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:43:32,994-Speed 1924.12 samples/sec Loss 1.4334 Epoch: 18 Global Step: 91250 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:43:59,626-Speed 1922.58 samples/sec Loss 1.4654 Epoch: 18 Global Step: 91300 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:44:25,785-Speed 1957.34 samples/sec Loss 1.4718 Epoch: 18 Global Step: 91350 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:44:52,659-Speed 1905.21 samples/sec Loss 1.4288 Epoch: 18 Global Step: 91400 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:45:19,268-Speed 1924.28 samples/sec Loss 1.4319 Epoch: 18 Global Step: 91450 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:45:45,682-Speed 1938.38 samples/sec Loss 1.4482 Epoch: 18 Global Step: 91500 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:46:12,443-Speed 1913.33 samples/sec Loss 1.4577 Epoch: 18 Global Step: 91550 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:46:39,098-Speed 1920.88 samples/sec Loss 1.4605 Epoch: 18 Global Step: 91600 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:47:05,578-Speed 1933.60 samples/sec Loss 1.4338 Epoch: 18 Global Step: 91650 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:47:32,034-Speed 1935.36 samples/sec Loss 1.4547 Epoch: 18 Global Step: 91700 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:47:58,252-Speed 1952.93 samples/sec Loss 1.4496 Epoch: 18 Global Step: 91750 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:48:24,674-Speed 1937.80 samples/sec Loss 1.4408 Epoch: 18 Global Step: 91800 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:48:51,339-Speed 1920.17 samples/sec Loss 1.4590 Epoch: 18 Global Step: 91850 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:49:17,823-Speed 1933.30 samples/sec Loss 1.4688 Epoch: 18 Global Step: 91900 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:49:44,083-Speed 1949.80 samples/sec Loss 1.4758 Epoch: 18 Global Step: 91950 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:50:10,972-Speed 1904.24 samples/sec Loss 1.4398 Epoch: 18 Global Step: 92000 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:51:14,030-[lfw][92000]XNorm: 22.612817 Training: 2021-03-15 23:51:14,031-[lfw][92000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-15 23:51:14,031-[lfw][92000]Accuracy-Highest: 0.99833 Training: 2021-03-15 23:52:26,181-[cfp_fp][92000]XNorm: 21.404146 Training: 2021-03-15 23:52:26,182-[cfp_fp][92000]Accuracy-Flip: 0.98914+-0.00287 Training: 2021-03-15 23:52:26,182-[cfp_fp][92000]Accuracy-Highest: 0.99000 Training: 2021-03-15 23:53:28,625-[agedb_30][92000]XNorm: 23.013588 Training: 2021-03-15 23:53:28,626-[agedb_30][92000]Accuracy-Flip: 0.98267+-0.00642 Training: 2021-03-15 23:53:28,626-[agedb_30][92000]Accuracy-Highest: 0.98400 Training: 2021-03-15 23:53:55,144-Speed 228.40 samples/sec Loss 1.4493 Epoch: 18 Global Step: 92050 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:54:21,642-Speed 1932.31 samples/sec Loss 1.4658 Epoch: 18 Global Step: 92100 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:54:47,950-Speed 1946.21 samples/sec Loss 1.4468 Epoch: 18 Global Step: 92150 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:55:14,344-Speed 1939.91 samples/sec Loss 1.4444 Epoch: 18 Global Step: 92200 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:55:40,863-Speed 1930.74 samples/sec Loss 1.4784 Epoch: 18 Global Step: 92250 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:56:07,479-Speed 1923.67 samples/sec Loss 1.4821 Epoch: 18 Global Step: 92300 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:56:33,613-Speed 1959.22 samples/sec Loss 1.4596 Epoch: 18 Global Step: 92350 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:57:00,003-Speed 1940.23 samples/sec Loss 1.4772 Epoch: 18 Global Step: 92400 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:57:26,318-Speed 1945.73 samples/sec Loss 1.5084 Epoch: 18 Global Step: 92450 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:57:52,556-Speed 1951.44 samples/sec Loss 1.4898 Epoch: 18 Global Step: 92500 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:58:18,701-Speed 1958.32 samples/sec Loss 1.4564 Epoch: 18 Global Step: 92550 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:58:45,291-Speed 1925.65 samples/sec Loss 1.4715 Epoch: 18 Global Step: 92600 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:59:11,900-Speed 1924.25 samples/sec Loss 1.4842 Epoch: 18 Global Step: 92650 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-15 23:59:38,251-Speed 1943.05 samples/sec Loss 1.4749 Epoch: 18 Global Step: 92700 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:00:04,475-Speed 1952.50 samples/sec Loss 1.5052 Epoch: 18 Global Step: 92750 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:00:30,866-Speed 1940.07 samples/sec Loss 1.5115 Epoch: 18 Global Step: 92800 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:00:57,630-Speed 1913.09 samples/sec Loss 1.4813 Epoch: 18 Global Step: 92850 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:01:23,816-Speed 1955.35 samples/sec Loss 1.4774 Epoch: 18 Global Step: 92900 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:01:50,222-Speed 1939.10 samples/sec Loss 1.4564 Epoch: 18 Global Step: 92950 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:02:16,686-Speed 1934.71 samples/sec Loss 1.4729 Epoch: 18 Global Step: 93000 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:02:43,330-Speed 1921.86 samples/sec Loss 1.4709 Epoch: 18 Global Step: 93050 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:03:10,052-Speed 1916.15 samples/sec Loss 1.5259 Epoch: 18 Global Step: 93100 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:03:36,959-Speed 1902.89 samples/sec Loss 1.4734 Epoch: 18 Global Step: 93150 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:04:03,309-Speed 1943.20 samples/sec Loss 1.5069 Epoch: 18 Global Step: 93200 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:04:29,964-Speed 1920.87 samples/sec Loss 1.5013 Epoch: 18 Global Step: 93250 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:04:56,625-Speed 1920.50 samples/sec Loss 1.5010 Epoch: 18 Global Step: 93300 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:05:22,959-Speed 1944.32 samples/sec Loss 1.4905 Epoch: 18 Global Step: 93350 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:05:49,362-Speed 1939.25 samples/sec Loss 1.4990 Epoch: 18 Global Step: 93400 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:06:15,821-Speed 1935.06 samples/sec Loss 1.4908 Epoch: 18 Global Step: 93450 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:06:42,596-Speed 1912.33 samples/sec Loss 1.5019 Epoch: 18 Global Step: 93500 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:07:08,793-Speed 1954.45 samples/sec Loss 1.4798 Epoch: 18 Global Step: 93550 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:07:35,524-Speed 1915.47 samples/sec Loss 1.4876 Epoch: 18 Global Step: 93600 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:08:01,897-Speed 1941.47 samples/sec Loss 1.5084 Epoch: 18 Global Step: 93650 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:08:28,230-Speed 1944.39 samples/sec Loss 1.5242 Epoch: 18 Global Step: 93700 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:08:54,974-Speed 1914.50 samples/sec Loss 1.4930 Epoch: 18 Global Step: 93750 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:09:21,016-Speed 1966.07 samples/sec Loss 1.4884 Epoch: 18 Global Step: 93800 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:09:47,721-Speed 1917.51 samples/sec Loss 1.5311 Epoch: 18 Global Step: 93850 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:10:13,626-Speed 1976.47 samples/sec Loss 1.5213 Epoch: 18 Global Step: 93900 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:10:39,971-Speed 1943.61 samples/sec Loss 1.4799 Epoch: 18 Global Step: 93950 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:11:06,429-Speed 1935.16 samples/sec Loss 1.5153 Epoch: 18 Global Step: 94000 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:12:08,673-[lfw][94000]XNorm: 22.821740 Training: 2021-03-16 00:12:08,673-[lfw][94000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 00:12:08,673-[lfw][94000]Accuracy-Highest: 0.99833 Training: 2021-03-16 00:13:22,323-[cfp_fp][94000]XNorm: 21.655495 Training: 2021-03-16 00:13:22,323-[cfp_fp][94000]Accuracy-Flip: 0.98986+-0.00186 Training: 2021-03-16 00:13:22,323-[cfp_fp][94000]Accuracy-Highest: 0.99000 Training: 2021-03-16 00:14:25,060-[agedb_30][94000]XNorm: 23.193808 Training: 2021-03-16 00:14:25,060-[agedb_30][94000]Accuracy-Flip: 0.98450+-0.00719 Training: 2021-03-16 00:14:25,060-[agedb_30][94000]Accuracy-Highest: 0.98450 Training: 2021-03-16 00:14:51,318-Speed 227.67 samples/sec Loss 1.5102 Epoch: 18 Global Step: 94050 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:15:17,873-Speed 1928.12 samples/sec Loss 1.4858 Epoch: 18 Global Step: 94100 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:15:44,151-Speed 1948.45 samples/sec Loss 1.5379 Epoch: 18 Global Step: 94150 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:16:10,174-Speed 1967.53 samples/sec Loss 1.4840 Epoch: 18 Global Step: 94200 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:16:37,144-Speed 1898.44 samples/sec Loss 1.5186 Epoch: 18 Global Step: 94250 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:17:03,429-Speed 1947.96 samples/sec Loss 1.5260 Epoch: 18 Global Step: 94300 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:17:29,777-Speed 1943.30 samples/sec Loss 1.4829 Epoch: 18 Global Step: 94350 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:17:56,372-Speed 1925.25 samples/sec Loss 1.4698 Epoch: 18 Global Step: 94400 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:18:22,917-Speed 1928.88 samples/sec Loss 1.5181 Epoch: 18 Global Step: 94450 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:18:50,078-Speed 1885.06 samples/sec Loss 1.5252 Epoch: 18 Global Step: 94500 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:19:16,646-Speed 1927.33 samples/sec Loss 1.5155 Epoch: 18 Global Step: 94550 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:19:43,505-Speed 1906.33 samples/sec Loss 1.5008 Epoch: 18 Global Step: 94600 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:20:10,274-Speed 1912.81 samples/sec Loss 1.5318 Epoch: 18 Global Step: 94650 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:20:41,091-Speed 1661.47 samples/sec Loss 1.2323 Epoch: 19 Global Step: 94700 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:21:07,826-Speed 1915.24 samples/sec Loss 1.2371 Epoch: 19 Global Step: 94750 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:21:34,360-Speed 1929.71 samples/sec Loss 1.1970 Epoch: 19 Global Step: 94800 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:22:00,427-Speed 1964.19 samples/sec Loss 1.1997 Epoch: 19 Global Step: 94850 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:22:26,526-Speed 1961.84 samples/sec Loss 1.2320 Epoch: 19 Global Step: 94900 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:22:53,168-Speed 1921.87 samples/sec Loss 1.2281 Epoch: 19 Global Step: 94950 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:23:19,318-Speed 1958.03 samples/sec Loss 1.1964 Epoch: 19 Global Step: 95000 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:23:45,684-Speed 1941.96 samples/sec Loss 1.2328 Epoch: 19 Global Step: 95050 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:24:12,253-Speed 1927.12 samples/sec Loss 1.2027 Epoch: 19 Global Step: 95100 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:24:38,624-Speed 1941.57 samples/sec Loss 1.2252 Epoch: 19 Global Step: 95150 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:25:05,025-Speed 1939.33 samples/sec Loss 1.2053 Epoch: 19 Global Step: 95200 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:25:31,605-Speed 1926.39 samples/sec Loss 1.2478 Epoch: 19 Global Step: 95250 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:25:58,029-Speed 1937.71 samples/sec Loss 1.2374 Epoch: 19 Global Step: 95300 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:26:24,040-Speed 1968.52 samples/sec Loss 1.2198 Epoch: 19 Global Step: 95350 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:26:50,289-Speed 1950.56 samples/sec Loss 1.2346 Epoch: 19 Global Step: 95400 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:27:16,600-Speed 1946.05 samples/sec Loss 1.2448 Epoch: 19 Global Step: 95450 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:27:43,236-Speed 1922.28 samples/sec Loss 1.2504 Epoch: 19 Global Step: 95500 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:28:09,522-Speed 1947.84 samples/sec Loss 1.2306 Epoch: 19 Global Step: 95550 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:28:36,071-Speed 1928.57 samples/sec Loss 1.2366 Epoch: 19 Global Step: 95600 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:29:02,497-Speed 1937.54 samples/sec Loss 1.2533 Epoch: 19 Global Step: 95650 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:29:28,511-Speed 1968.28 samples/sec Loss 1.2354 Epoch: 19 Global Step: 95700 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:29:55,026-Speed 1931.02 samples/sec Loss 1.2843 Epoch: 19 Global Step: 95750 Fp16 Grad Scale: 16384 Required: 5 hours Training: 2021-03-16 00:30:21,393-Speed 1941.90 samples/sec Loss 1.2103 Epoch: 19 Global Step: 95800 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:30:47,572-Speed 1955.76 samples/sec Loss 1.2915 Epoch: 19 Global Step: 95850 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:31:14,183-Speed 1924.08 samples/sec Loss 1.2482 Epoch: 19 Global Step: 95900 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:31:40,582-Speed 1939.54 samples/sec Loss 1.2802 Epoch: 19 Global Step: 95950 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:32:07,072-Speed 1932.86 samples/sec Loss 1.2424 Epoch: 19 Global Step: 96000 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:33:10,152-[lfw][96000]XNorm: 21.938029 Training: 2021-03-16 00:33:10,152-[lfw][96000]Accuracy-Flip: 0.99850+-0.00189 Training: 2021-03-16 00:33:10,152-[lfw][96000]Accuracy-Highest: 0.99850 Training: 2021-03-16 00:34:21,307-[cfp_fp][96000]XNorm: 21.104041 Training: 2021-03-16 00:34:21,308-[cfp_fp][96000]Accuracy-Flip: 0.99000+-0.00247 Training: 2021-03-16 00:34:21,308-[cfp_fp][96000]Accuracy-Highest: 0.99000 Training: 2021-03-16 00:35:22,251-[agedb_30][96000]XNorm: 22.507495 Training: 2021-03-16 00:35:22,251-[agedb_30][96000]Accuracy-Flip: 0.98467+-0.00745 Training: 2021-03-16 00:35:22,251-[agedb_30][96000]Accuracy-Highest: 0.98467 Training: 2021-03-16 00:35:48,342-Speed 231.39 samples/sec Loss 1.2505 Epoch: 19 Global Step: 96050 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:36:14,610-Speed 1949.17 samples/sec Loss 1.2810 Epoch: 19 Global Step: 96100 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:36:41,037-Speed 1937.52 samples/sec Loss 1.3044 Epoch: 19 Global Step: 96150 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:37:07,713-Speed 1919.39 samples/sec Loss 1.2805 Epoch: 19 Global Step: 96200 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:37:34,225-Speed 1931.24 samples/sec Loss 1.2503 Epoch: 19 Global Step: 96250 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:38:00,917-Speed 1918.25 samples/sec Loss 1.2841 Epoch: 19 Global Step: 96300 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:38:27,275-Speed 1942.56 samples/sec Loss 1.2992 Epoch: 19 Global Step: 96350 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:38:52,408-Speed 2037.29 samples/sec Loss 1.2413 Epoch: 19 Global Step: 96400 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:39:18,817-Speed 1938.73 samples/sec Loss 1.2867 Epoch: 19 Global Step: 96450 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:39:45,630-Speed 1909.62 samples/sec Loss 1.3034 Epoch: 19 Global Step: 96500 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:40:11,801-Speed 1956.41 samples/sec Loss 1.2733 Epoch: 19 Global Step: 96550 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:40:38,369-Speed 1927.15 samples/sec Loss 1.2886 Epoch: 19 Global Step: 96600 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:41:04,733-Speed 1942.12 samples/sec Loss 1.2623 Epoch: 19 Global Step: 96650 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:41:31,321-Speed 1925.75 samples/sec Loss 1.2814 Epoch: 19 Global Step: 96700 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:41:57,843-Speed 1930.58 samples/sec Loss 1.3189 Epoch: 19 Global Step: 96750 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:42:23,836-Speed 1969.98 samples/sec Loss 1.2900 Epoch: 19 Global Step: 96800 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:42:50,172-Speed 1944.17 samples/sec Loss 1.2706 Epoch: 19 Global Step: 96850 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:43:16,747-Speed 1926.69 samples/sec Loss 1.3073 Epoch: 19 Global Step: 96900 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:43:42,893-Speed 1958.24 samples/sec Loss 1.3089 Epoch: 19 Global Step: 96950 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:44:09,034-Speed 1958.72 samples/sec Loss 1.2992 Epoch: 19 Global Step: 97000 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:44:35,284-Speed 1950.55 samples/sec Loss 1.3050 Epoch: 19 Global Step: 97050 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:45:01,538-Speed 1950.26 samples/sec Loss 1.3110 Epoch: 19 Global Step: 97100 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:45:27,804-Speed 1949.34 samples/sec Loss 1.3147 Epoch: 19 Global Step: 97150 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:45:54,042-Speed 1951.50 samples/sec Loss 1.3102 Epoch: 19 Global Step: 97200 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:46:20,446-Speed 1939.17 samples/sec Loss 1.2973 Epoch: 19 Global Step: 97250 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:46:46,947-Speed 1932.06 samples/sec Loss 1.3022 Epoch: 19 Global Step: 97300 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:47:13,218-Speed 1948.97 samples/sec Loss 1.3054 Epoch: 19 Global Step: 97350 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:47:40,018-Speed 1910.60 samples/sec Loss 1.2992 Epoch: 19 Global Step: 97400 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:48:06,637-Speed 1923.49 samples/sec Loss 1.3195 Epoch: 19 Global Step: 97450 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:48:33,036-Speed 1939.68 samples/sec Loss 1.3506 Epoch: 19 Global Step: 97500 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:48:59,741-Speed 1917.28 samples/sec Loss 1.3107 Epoch: 19 Global Step: 97550 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:49:26,409-Speed 1919.97 samples/sec Loss 1.3012 Epoch: 19 Global Step: 97600 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:49:52,844-Speed 1936.88 samples/sec Loss 1.3289 Epoch: 19 Global Step: 97650 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:50:19,458-Speed 1923.88 samples/sec Loss 1.3455 Epoch: 19 Global Step: 97700 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:50:45,758-Speed 1946.86 samples/sec Loss 1.3316 Epoch: 19 Global Step: 97750 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:51:11,683-Speed 1975.07 samples/sec Loss 1.3164 Epoch: 19 Global Step: 97800 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:51:37,972-Speed 1947.62 samples/sec Loss 1.3260 Epoch: 19 Global Step: 97850 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:52:04,093-Speed 1960.24 samples/sec Loss 1.3184 Epoch: 19 Global Step: 97900 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:52:30,595-Speed 1931.93 samples/sec Loss 1.3660 Epoch: 19 Global Step: 97950 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:52:57,155-Speed 1927.82 samples/sec Loss 1.3263 Epoch: 19 Global Step: 98000 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:53:59,215-[lfw][98000]XNorm: 22.649005 Training: 2021-03-16 00:53:59,215-[lfw][98000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 00:53:59,215-[lfw][98000]Accuracy-Highest: 0.99850 Training: 2021-03-16 00:55:10,727-[cfp_fp][98000]XNorm: 21.563274 Training: 2021-03-16 00:55:10,727-[cfp_fp][98000]Accuracy-Flip: 0.98957+-0.00239 Training: 2021-03-16 00:55:10,727-[cfp_fp][98000]Accuracy-Highest: 0.99000 Training: 2021-03-16 00:56:13,678-[agedb_30][98000]XNorm: 23.107395 Training: 2021-03-16 00:56:13,679-[agedb_30][98000]Accuracy-Flip: 0.98450+-0.00742 Training: 2021-03-16 00:56:13,679-[agedb_30][98000]Accuracy-Highest: 0.98467 Training: 2021-03-16 00:56:39,646-Speed 230.12 samples/sec Loss 1.3679 Epoch: 19 Global Step: 98050 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:57:05,827-Speed 1955.70 samples/sec Loss 1.3410 Epoch: 19 Global Step: 98100 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:57:31,666-Speed 1981.54 samples/sec Loss 1.3430 Epoch: 19 Global Step: 98150 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:57:58,159-Speed 1932.67 samples/sec Loss 1.3405 Epoch: 19 Global Step: 98200 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:58:24,268-Speed 1961.04 samples/sec Loss 1.3397 Epoch: 19 Global Step: 98250 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:58:50,802-Speed 1929.70 samples/sec Loss 1.3369 Epoch: 19 Global Step: 98300 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:59:17,428-Speed 1922.99 samples/sec Loss 1.3485 Epoch: 19 Global Step: 98350 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 00:59:43,600-Speed 1956.35 samples/sec Loss 1.3361 Epoch: 19 Global Step: 98400 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:00:09,939-Speed 1943.94 samples/sec Loss 1.3582 Epoch: 19 Global Step: 98450 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:00:36,984-Speed 1893.18 samples/sec Loss 1.3354 Epoch: 19 Global Step: 98500 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:01:03,431-Speed 1936.03 samples/sec Loss 1.3588 Epoch: 19 Global Step: 98550 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:01:29,725-Speed 1947.25 samples/sec Loss 1.3466 Epoch: 19 Global Step: 98600 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:01:55,947-Speed 1952.65 samples/sec Loss 1.3692 Epoch: 19 Global Step: 98650 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:02:22,159-Speed 1953.37 samples/sec Loss 1.3705 Epoch: 19 Global Step: 98700 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:02:48,308-Speed 1958.04 samples/sec Loss 1.3646 Epoch: 19 Global Step: 98750 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:03:14,640-Speed 1944.66 samples/sec Loss 1.3847 Epoch: 19 Global Step: 98800 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:03:40,989-Speed 1943.16 samples/sec Loss 1.3640 Epoch: 19 Global Step: 98850 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:04:07,347-Speed 1942.52 samples/sec Loss 1.3586 Epoch: 19 Global Step: 98900 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:04:33,742-Speed 1939.88 samples/sec Loss 1.3599 Epoch: 19 Global Step: 98950 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:05:00,143-Speed 1939.34 samples/sec Loss 1.3802 Epoch: 19 Global Step: 99000 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:05:26,994-Speed 1906.92 samples/sec Loss 1.3689 Epoch: 19 Global Step: 99050 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:05:53,140-Speed 1958.32 samples/sec Loss 1.3863 Epoch: 19 Global Step: 99100 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:06:19,285-Speed 1958.33 samples/sec Loss 1.3628 Epoch: 19 Global Step: 99150 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:06:45,985-Speed 1917.68 samples/sec Loss 1.3752 Epoch: 19 Global Step: 99200 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:07:12,749-Speed 1913.07 samples/sec Loss 1.3801 Epoch: 19 Global Step: 99250 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:07:39,254-Speed 1931.78 samples/sec Loss 1.3568 Epoch: 19 Global Step: 99300 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:08:05,789-Speed 1929.62 samples/sec Loss 1.4030 Epoch: 19 Global Step: 99350 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:08:32,115-Speed 1944.96 samples/sec Loss 1.4010 Epoch: 19 Global Step: 99400 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:08:58,955-Speed 1907.68 samples/sec Loss 1.3670 Epoch: 19 Global Step: 99450 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:09:25,748-Speed 1911.09 samples/sec Loss 1.3548 Epoch: 19 Global Step: 99500 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:09:51,978-Speed 1952.01 samples/sec Loss 1.3816 Epoch: 19 Global Step: 99550 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:10:18,220-Speed 1951.10 samples/sec Loss 1.3610 Epoch: 19 Global Step: 99600 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:10:49,824-Speed 1620.08 samples/sec Loss 1.2899 Epoch: 20 Global Step: 99650 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:11:16,136-Speed 1945.95 samples/sec Loss 1.0420 Epoch: 20 Global Step: 99700 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:11:42,877-Speed 1914.71 samples/sec Loss 1.0701 Epoch: 20 Global Step: 99750 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:12:09,222-Speed 1943.63 samples/sec Loss 1.0498 Epoch: 20 Global Step: 99800 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:12:35,297-Speed 1963.62 samples/sec Loss 1.0847 Epoch: 20 Global Step: 99850 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:13:01,750-Speed 1935.58 samples/sec Loss 1.0741 Epoch: 20 Global Step: 99900 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:13:28,142-Speed 1940.02 samples/sec Loss 1.1084 Epoch: 20 Global Step: 99950 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:13:54,772-Speed 1922.73 samples/sec Loss 1.1002 Epoch: 20 Global Step: 100000 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:14:56,956-[lfw][100000]XNorm: 22.724169 Training: 2021-03-16 01:14:56,956-[lfw][100000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 01:14:56,956-[lfw][100000]Accuracy-Highest: 0.99850 Training: 2021-03-16 01:16:10,583-[cfp_fp][100000]XNorm: 21.700528 Training: 2021-03-16 01:16:10,583-[cfp_fp][100000]Accuracy-Flip: 0.98871+-0.00243 Training: 2021-03-16 01:16:10,583-[cfp_fp][100000]Accuracy-Highest: 0.99000 Training: 2021-03-16 01:17:14,135-[agedb_30][100000]XNorm: 22.990454 Training: 2021-03-16 01:17:14,135-[agedb_30][100000]Accuracy-Flip: 0.98450+-0.00633 Training: 2021-03-16 01:17:14,136-[agedb_30][100000]Accuracy-Highest: 0.98467 Training: 2021-03-16 01:17:40,333-Speed 226.99 samples/sec Loss 1.0680 Epoch: 20 Global Step: 100050 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:18:07,083-Speed 1914.10 samples/sec Loss 1.0940 Epoch: 20 Global Step: 100100 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:18:32,858-Speed 1986.50 samples/sec Loss 1.0829 Epoch: 20 Global Step: 100150 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:18:59,319-Speed 1935.06 samples/sec Loss 1.1024 Epoch: 20 Global Step: 100200 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:19:25,815-Speed 1932.41 samples/sec Loss 1.0910 Epoch: 20 Global Step: 100250 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:19:52,311-Speed 1932.45 samples/sec Loss 1.1189 Epoch: 20 Global Step: 100300 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:20:18,540-Speed 1952.12 samples/sec Loss 1.1188 Epoch: 20 Global Step: 100350 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:20:44,700-Speed 1957.36 samples/sec Loss 1.0867 Epoch: 20 Global Step: 100400 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:21:11,245-Speed 1928.81 samples/sec Loss 1.1048 Epoch: 20 Global Step: 100450 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:21:37,753-Speed 1931.60 samples/sec Loss 1.1265 Epoch: 20 Global Step: 100500 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:22:04,096-Speed 1943.65 samples/sec Loss 1.1137 Epoch: 20 Global Step: 100550 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:22:30,150-Speed 1965.22 samples/sec Loss 1.1230 Epoch: 20 Global Step: 100600 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:22:56,383-Speed 1951.80 samples/sec Loss 1.1379 Epoch: 20 Global Step: 100650 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:23:23,123-Speed 1914.73 samples/sec Loss 1.0980 Epoch: 20 Global Step: 100700 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:23:49,451-Speed 1944.81 samples/sec Loss 1.1013 Epoch: 20 Global Step: 100750 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:24:15,802-Speed 1943.07 samples/sec Loss 1.1433 Epoch: 20 Global Step: 100800 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:24:42,425-Speed 1923.17 samples/sec Loss 1.1575 Epoch: 20 Global Step: 100850 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:25:08,806-Speed 1940.88 samples/sec Loss 1.1250 Epoch: 20 Global Step: 100900 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:25:35,296-Speed 1932.99 samples/sec Loss 1.1552 Epoch: 20 Global Step: 100950 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:26:01,573-Speed 1948.54 samples/sec Loss 1.1193 Epoch: 20 Global Step: 101000 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:26:28,118-Speed 1928.85 samples/sec Loss 1.1449 Epoch: 20 Global Step: 101050 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:26:54,593-Speed 1933.94 samples/sec Loss 1.1362 Epoch: 20 Global Step: 101100 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:27:20,774-Speed 1955.67 samples/sec Loss 1.1440 Epoch: 20 Global Step: 101150 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:27:46,870-Speed 1962.07 samples/sec Loss 1.1817 Epoch: 20 Global Step: 101200 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:28:13,379-Speed 1931.44 samples/sec Loss 1.1481 Epoch: 20 Global Step: 101250 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:28:39,935-Speed 1928.10 samples/sec Loss 1.1655 Epoch: 20 Global Step: 101300 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:29:06,348-Speed 1938.48 samples/sec Loss 1.1440 Epoch: 20 Global Step: 101350 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:29:33,070-Speed 1916.05 samples/sec Loss 1.1551 Epoch: 20 Global Step: 101400 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:29:59,398-Speed 1944.79 samples/sec Loss 1.1687 Epoch: 20 Global Step: 101450 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:30:25,518-Speed 1960.24 samples/sec Loss 1.1515 Epoch: 20 Global Step: 101500 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:30:51,517-Speed 1969.40 samples/sec Loss 1.1671 Epoch: 20 Global Step: 101550 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:31:17,634-Speed 1960.44 samples/sec Loss 1.2072 Epoch: 20 Global Step: 101600 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:31:44,417-Speed 1911.73 samples/sec Loss 1.1895 Epoch: 20 Global Step: 101650 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:32:11,106-Speed 1918.49 samples/sec Loss 1.1637 Epoch: 20 Global Step: 101700 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:32:37,467-Speed 1942.31 samples/sec Loss 1.1804 Epoch: 20 Global Step: 101750 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:33:03,491-Speed 1967.43 samples/sec Loss 1.1962 Epoch: 20 Global Step: 101800 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:33:30,003-Speed 1931.28 samples/sec Loss 1.1920 Epoch: 20 Global Step: 101850 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:33:56,324-Speed 1945.31 samples/sec Loss 1.1845 Epoch: 20 Global Step: 101900 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:34:22,647-Speed 1945.07 samples/sec Loss 1.2154 Epoch: 20 Global Step: 101950 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:34:49,097-Speed 1935.82 samples/sec Loss 1.2059 Epoch: 20 Global Step: 102000 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:35:51,671-[lfw][102000]XNorm: 22.509972 Training: 2021-03-16 01:35:51,671-[lfw][102000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 01:35:51,671-[lfw][102000]Accuracy-Highest: 0.99850 Training: 2021-03-16 01:37:03,538-[cfp_fp][102000]XNorm: 21.525276 Training: 2021-03-16 01:37:03,538-[cfp_fp][102000]Accuracy-Flip: 0.99057+-0.00232 Training: 2021-03-16 01:37:03,538-[cfp_fp][102000]Accuracy-Highest: 0.99057 Training: 2021-03-16 01:38:06,968-[agedb_30][102000]XNorm: 22.905842 Training: 2021-03-16 01:38:06,968-[agedb_30][102000]Accuracy-Flip: 0.98333+-0.00734 Training: 2021-03-16 01:38:06,968-[agedb_30][102000]Accuracy-Highest: 0.98467 Training: 2021-03-16 01:38:33,502-Speed 228.16 samples/sec Loss 1.1801 Epoch: 20 Global Step: 102050 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:38:59,661-Speed 1957.33 samples/sec Loss 1.2155 Epoch: 20 Global Step: 102100 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:39:25,989-Speed 1944.71 samples/sec Loss 1.2176 Epoch: 20 Global Step: 102150 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:39:52,392-Speed 1939.29 samples/sec Loss 1.2034 Epoch: 20 Global Step: 102200 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:40:19,160-Speed 1912.78 samples/sec Loss 1.2097 Epoch: 20 Global Step: 102250 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:40:45,602-Speed 1936.40 samples/sec Loss 1.2175 Epoch: 20 Global Step: 102300 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:41:12,326-Speed 1915.96 samples/sec Loss 1.1951 Epoch: 20 Global Step: 102350 Fp16 Grad Scale: 16384 Required: 4 hours Training: 2021-03-16 01:41:38,499-Speed 1956.42 samples/sec Loss 1.2214 Epoch: 20 Global Step: 102400 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:42:04,810-Speed 1946.04 samples/sec Loss 1.2344 Epoch: 20 Global Step: 102450 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:42:30,716-Speed 1976.48 samples/sec Loss 1.2070 Epoch: 20 Global Step: 102500 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:42:57,351-Speed 1922.31 samples/sec Loss 1.2254 Epoch: 20 Global Step: 102550 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:43:23,739-Speed 1940.32 samples/sec Loss 1.1961 Epoch: 20 Global Step: 102600 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:43:50,012-Speed 1948.80 samples/sec Loss 1.1976 Epoch: 20 Global Step: 102650 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:44:16,305-Speed 1947.33 samples/sec Loss 1.2215 Epoch: 20 Global Step: 102700 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:44:42,511-Speed 1953.87 samples/sec Loss 1.2021 Epoch: 20 Global Step: 102750 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:45:08,892-Speed 1940.81 samples/sec Loss 1.2289 Epoch: 20 Global Step: 102800 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:45:35,481-Speed 1925.67 samples/sec Loss 1.2509 Epoch: 20 Global Step: 102850 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:46:01,780-Speed 1946.95 samples/sec Loss 1.2432 Epoch: 20 Global Step: 102900 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:46:28,370-Speed 1925.58 samples/sec Loss 1.2417 Epoch: 20 Global Step: 102950 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:46:54,914-Speed 1928.94 samples/sec Loss 1.2256 Epoch: 20 Global Step: 103000 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:47:21,467-Speed 1928.30 samples/sec Loss 1.2443 Epoch: 20 Global Step: 103050 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:47:47,708-Speed 1951.16 samples/sec Loss 1.2502 Epoch: 20 Global Step: 103100 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:48:13,779-Speed 1963.97 samples/sec Loss 1.2214 Epoch: 20 Global Step: 103150 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:48:39,997-Speed 1953.00 samples/sec Loss 1.2045 Epoch: 20 Global Step: 103200 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:49:06,181-Speed 1955.43 samples/sec Loss 1.2469 Epoch: 20 Global Step: 103250 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:49:32,416-Speed 1951.79 samples/sec Loss 1.2604 Epoch: 20 Global Step: 103300 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:49:59,673-Speed 1878.43 samples/sec Loss 1.2669 Epoch: 20 Global Step: 103350 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:50:26,124-Speed 1935.74 samples/sec Loss 1.2364 Epoch: 20 Global Step: 103400 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:50:52,317-Speed 1954.85 samples/sec Loss 1.2512 Epoch: 20 Global Step: 103450 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:51:18,617-Speed 1946.84 samples/sec Loss 1.2588 Epoch: 20 Global Step: 103500 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:51:45,031-Speed 1938.44 samples/sec Loss 1.2679 Epoch: 20 Global Step: 103550 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:52:11,794-Speed 1913.16 samples/sec Loss 1.2517 Epoch: 20 Global Step: 103600 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:52:38,118-Speed 1945.01 samples/sec Loss 1.2381 Epoch: 20 Global Step: 103650 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:53:04,297-Speed 1955.82 samples/sec Loss 1.2852 Epoch: 20 Global Step: 103700 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:53:30,839-Speed 1929.26 samples/sec Loss 1.2628 Epoch: 20 Global Step: 103750 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:53:57,152-Speed 1945.85 samples/sec Loss 1.2749 Epoch: 20 Global Step: 103800 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:54:23,783-Speed 1922.64 samples/sec Loss 1.2708 Epoch: 20 Global Step: 103850 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:54:50,181-Speed 1939.57 samples/sec Loss 1.2848 Epoch: 20 Global Step: 103900 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:55:16,425-Speed 1951.01 samples/sec Loss 1.2612 Epoch: 20 Global Step: 103950 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:55:42,978-Speed 1928.29 samples/sec Loss 1.2778 Epoch: 20 Global Step: 104000 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:56:46,308-[lfw][104000]XNorm: 22.394930 Training: 2021-03-16 01:56:46,308-[lfw][104000]Accuracy-Flip: 0.99817+-0.00241 Training: 2021-03-16 01:56:46,308-[lfw][104000]Accuracy-Highest: 0.99850 Training: 2021-03-16 01:58:00,220-[cfp_fp][104000]XNorm: 21.430063 Training: 2021-03-16 01:58:00,220-[cfp_fp][104000]Accuracy-Flip: 0.98929+-0.00265 Training: 2021-03-16 01:58:00,221-[cfp_fp][104000]Accuracy-Highest: 0.99057 Training: 2021-03-16 01:59:03,054-[agedb_30][104000]XNorm: 22.710899 Training: 2021-03-16 01:59:03,054-[agedb_30][104000]Accuracy-Flip: 0.98367+-0.00748 Training: 2021-03-16 01:59:03,054-[agedb_30][104000]Accuracy-Highest: 0.98467 Training: 2021-03-16 01:59:29,714-Speed 225.81 samples/sec Loss 1.2777 Epoch: 20 Global Step: 104050 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 01:59:56,066-Speed 1942.92 samples/sec Loss 1.2891 Epoch: 20 Global Step: 104100 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:00:22,177-Speed 1960.93 samples/sec Loss 1.2536 Epoch: 20 Global Step: 104150 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:00:48,830-Speed 1921.02 samples/sec Loss 1.2619 Epoch: 20 Global Step: 104200 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:01:15,719-Speed 1904.21 samples/sec Loss 1.2578 Epoch: 20 Global Step: 104250 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:01:41,656-Speed 1974.12 samples/sec Loss 1.2688 Epoch: 20 Global Step: 104300 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:02:08,165-Speed 1931.60 samples/sec Loss 1.3002 Epoch: 20 Global Step: 104350 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:02:34,912-Speed 1914.29 samples/sec Loss 1.2695 Epoch: 20 Global Step: 104400 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:03:01,017-Speed 1961.39 samples/sec Loss 1.3128 Epoch: 20 Global Step: 104450 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:03:27,575-Speed 1927.89 samples/sec Loss 1.3264 Epoch: 20 Global Step: 104500 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:03:53,854-Speed 1948.39 samples/sec Loss 1.2743 Epoch: 20 Global Step: 104550 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:04:20,200-Speed 1943.42 samples/sec Loss 1.2789 Epoch: 20 Global Step: 104600 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:04:51,276-Speed 1647.65 samples/sec Loss 1.1170 Epoch: 21 Global Step: 104650 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:05:18,140-Speed 1905.93 samples/sec Loss 0.9457 Epoch: 21 Global Step: 104700 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:05:44,177-Speed 1966.46 samples/sec Loss 0.9349 Epoch: 21 Global Step: 104750 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:06:10,370-Speed 1954.81 samples/sec Loss 0.9173 Epoch: 21 Global Step: 104800 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:06:36,598-Speed 1952.23 samples/sec Loss 0.9112 Epoch: 21 Global Step: 104850 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:07:02,805-Speed 1953.82 samples/sec Loss 0.9199 Epoch: 21 Global Step: 104900 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:07:29,356-Speed 1928.40 samples/sec Loss 0.9274 Epoch: 21 Global Step: 104950 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:07:56,149-Speed 1911.03 samples/sec Loss 0.9176 Epoch: 21 Global Step: 105000 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:08:22,494-Speed 1943.51 samples/sec Loss 0.9113 Epoch: 21 Global Step: 105050 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:08:48,866-Speed 1941.54 samples/sec Loss 0.9019 Epoch: 21 Global Step: 105100 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:09:15,297-Speed 1937.20 samples/sec Loss 0.9150 Epoch: 21 Global Step: 105150 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:09:42,033-Speed 1915.14 samples/sec Loss 0.8952 Epoch: 21 Global Step: 105200 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:10:08,283-Speed 1950.65 samples/sec Loss 0.9082 Epoch: 21 Global Step: 105250 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:10:34,168-Speed 1977.98 samples/sec Loss 0.9158 Epoch: 21 Global Step: 105300 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:11:00,592-Speed 1937.70 samples/sec Loss 0.8947 Epoch: 21 Global Step: 105350 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:11:27,078-Speed 1933.14 samples/sec Loss 0.9079 Epoch: 21 Global Step: 105400 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:11:53,684-Speed 1924.42 samples/sec Loss 0.8957 Epoch: 21 Global Step: 105450 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:12:20,062-Speed 1941.14 samples/sec Loss 0.8972 Epoch: 21 Global Step: 105500 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:12:46,482-Speed 1937.94 samples/sec Loss 0.8987 Epoch: 21 Global Step: 105550 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:13:12,841-Speed 1942.52 samples/sec Loss 0.9037 Epoch: 21 Global Step: 105600 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:13:39,165-Speed 1945.12 samples/sec Loss 0.9049 Epoch: 21 Global Step: 105650 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:14:06,193-Speed 1894.37 samples/sec Loss 0.8845 Epoch: 21 Global Step: 105700 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:14:32,548-Speed 1942.79 samples/sec Loss 0.8920 Epoch: 21 Global Step: 105750 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:14:59,006-Speed 1935.21 samples/sec Loss 0.9066 Epoch: 21 Global Step: 105800 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:15:26,179-Speed 1884.28 samples/sec Loss 0.8799 Epoch: 21 Global Step: 105850 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:15:52,841-Speed 1920.39 samples/sec Loss 0.8848 Epoch: 21 Global Step: 105900 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:16:19,441-Speed 1924.85 samples/sec Loss 0.9118 Epoch: 21 Global Step: 105950 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:16:46,042-Speed 1924.81 samples/sec Loss 0.8926 Epoch: 21 Global Step: 106000 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:17:49,542-[lfw][106000]XNorm: 22.330657 Training: 2021-03-16 02:17:49,542-[lfw][106000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 02:17:49,542-[lfw][106000]Accuracy-Highest: 0.99850 Training: 2021-03-16 02:19:01,855-[cfp_fp][106000]XNorm: 21.574163 Training: 2021-03-16 02:19:01,855-[cfp_fp][106000]Accuracy-Flip: 0.99029+-0.00219 Training: 2021-03-16 02:19:01,855-[cfp_fp][106000]Accuracy-Highest: 0.99057 Training: 2021-03-16 02:20:03,338-[agedb_30][106000]XNorm: 22.756268 Training: 2021-03-16 02:20:03,338-[agedb_30][106000]Accuracy-Flip: 0.98367+-0.00763 Training: 2021-03-16 02:20:03,339-[agedb_30][106000]Accuracy-Highest: 0.98467 Training: 2021-03-16 02:20:29,334-Speed 229.30 samples/sec Loss 0.9267 Epoch: 21 Global Step: 106050 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:20:56,006-Speed 1919.69 samples/sec Loss 0.9220 Epoch: 21 Global Step: 106100 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:21:22,269-Speed 1949.55 samples/sec Loss 0.8794 Epoch: 21 Global Step: 106150 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:21:48,677-Speed 1938.95 samples/sec Loss 0.8959 Epoch: 21 Global Step: 106200 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:22:14,584-Speed 1976.30 samples/sec Loss 0.8856 Epoch: 21 Global Step: 106250 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:22:41,650-Speed 1891.74 samples/sec Loss 0.8932 Epoch: 21 Global Step: 106300 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:23:07,970-Speed 1945.36 samples/sec Loss 0.8993 Epoch: 21 Global Step: 106350 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:23:34,304-Speed 1944.28 samples/sec Loss 0.8948 Epoch: 21 Global Step: 106400 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:24:00,703-Speed 1939.56 samples/sec Loss 0.8864 Epoch: 21 Global Step: 106450 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:24:26,907-Speed 1954.14 samples/sec Loss 0.9023 Epoch: 21 Global Step: 106500 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:24:53,575-Speed 1919.93 samples/sec Loss 0.8854 Epoch: 21 Global Step: 106550 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:25:20,416-Speed 1907.63 samples/sec Loss 0.9131 Epoch: 21 Global Step: 106600 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:25:46,938-Speed 1930.49 samples/sec Loss 0.9053 Epoch: 21 Global Step: 106650 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:26:13,546-Speed 1924.29 samples/sec Loss 0.8947 Epoch: 21 Global Step: 106700 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:26:39,439-Speed 1977.48 samples/sec Loss 0.9002 Epoch: 21 Global Step: 106750 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:27:05,560-Speed 1960.13 samples/sec Loss 0.8845 Epoch: 21 Global Step: 106800 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:27:31,427-Speed 1979.43 samples/sec Loss 0.9046 Epoch: 21 Global Step: 106850 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:27:58,015-Speed 1925.77 samples/sec Loss 0.8973 Epoch: 21 Global Step: 106900 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:28:24,615-Speed 1924.84 samples/sec Loss 0.9076 Epoch: 21 Global Step: 106950 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:28:50,801-Speed 1955.32 samples/sec Loss 0.9065 Epoch: 21 Global Step: 107000 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:29:17,098-Speed 1947.06 samples/sec Loss 0.8782 Epoch: 21 Global Step: 107050 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:29:43,305-Speed 1953.72 samples/sec Loss 0.8828 Epoch: 21 Global Step: 107100 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:30:09,525-Speed 1952.80 samples/sec Loss 0.8996 Epoch: 21 Global Step: 107150 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:30:35,767-Speed 1951.12 samples/sec Loss 0.9060 Epoch: 21 Global Step: 107200 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:31:02,096-Speed 1944.80 samples/sec Loss 0.8828 Epoch: 21 Global Step: 107250 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:31:28,303-Speed 1953.83 samples/sec Loss 0.9016 Epoch: 21 Global Step: 107300 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:31:54,750-Speed 1936.07 samples/sec Loss 0.8927 Epoch: 21 Global Step: 107350 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:32:21,547-Speed 1910.74 samples/sec Loss 0.8751 Epoch: 21 Global Step: 107400 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:32:47,579-Speed 1966.87 samples/sec Loss 0.8706 Epoch: 21 Global Step: 107450 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:33:14,487-Speed 1902.84 samples/sec Loss 0.8884 Epoch: 21 Global Step: 107500 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:33:40,812-Speed 1945.04 samples/sec Loss 0.8910 Epoch: 21 Global Step: 107550 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:34:07,079-Speed 1949.28 samples/sec Loss 0.8918 Epoch: 21 Global Step: 107600 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:34:33,528-Speed 1935.87 samples/sec Loss 0.9019 Epoch: 21 Global Step: 107650 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:34:59,903-Speed 1941.28 samples/sec Loss 0.8818 Epoch: 21 Global Step: 107700 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:35:26,167-Speed 1949.53 samples/sec Loss 0.9146 Epoch: 21 Global Step: 107750 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:35:52,473-Speed 1946.39 samples/sec Loss 0.8832 Epoch: 21 Global Step: 107800 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:36:19,143-Speed 1920.00 samples/sec Loss 0.8866 Epoch: 21 Global Step: 107850 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:36:45,685-Speed 1929.09 samples/sec Loss 0.8858 Epoch: 21 Global Step: 107900 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:37:11,743-Speed 1964.91 samples/sec Loss 0.8799 Epoch: 21 Global Step: 107950 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:37:38,496-Speed 1913.86 samples/sec Loss 0.8694 Epoch: 21 Global Step: 108000 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:38:40,237-[lfw][108000]XNorm: 22.441265 Training: 2021-03-16 02:38:40,238-[lfw][108000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 02:38:40,238-[lfw][108000]Accuracy-Highest: 0.99850 Training: 2021-03-16 02:39:52,525-[cfp_fp][108000]XNorm: 21.690724 Training: 2021-03-16 02:39:52,525-[cfp_fp][108000]Accuracy-Flip: 0.99014+-0.00251 Training: 2021-03-16 02:39:52,525-[cfp_fp][108000]Accuracy-Highest: 0.99057 Training: 2021-03-16 02:40:55,734-[agedb_30][108000]XNorm: 22.879818 Training: 2021-03-16 02:40:55,734-[agedb_30][108000]Accuracy-Flip: 0.98400+-0.00739 Training: 2021-03-16 02:40:55,734-[agedb_30][108000]Accuracy-Highest: 0.98467 Training: 2021-03-16 02:41:22,041-Speed 229.04 samples/sec Loss 0.8538 Epoch: 21 Global Step: 108050 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:41:48,772-Speed 1915.51 samples/sec Loss 0.8801 Epoch: 21 Global Step: 108100 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:42:14,491-Speed 1990.74 samples/sec Loss 0.9040 Epoch: 21 Global Step: 108150 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:42:41,423-Speed 1901.20 samples/sec Loss 0.8873 Epoch: 21 Global Step: 108200 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:43:07,971-Speed 1928.60 samples/sec Loss 0.8916 Epoch: 21 Global Step: 108250 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:43:34,412-Speed 1936.48 samples/sec Loss 0.8904 Epoch: 21 Global Step: 108300 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:44:00,467-Speed 1965.10 samples/sec Loss 0.8977 Epoch: 21 Global Step: 108350 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:44:26,886-Speed 1938.08 samples/sec Loss 0.8747 Epoch: 21 Global Step: 108400 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:44:53,228-Speed 1943.72 samples/sec Loss 0.9068 Epoch: 21 Global Step: 108450 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:45:19,204-Speed 1971.08 samples/sec Loss 0.9163 Epoch: 21 Global Step: 108500 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:45:45,431-Speed 1952.31 samples/sec Loss 0.8736 Epoch: 21 Global Step: 108550 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:46:11,594-Speed 1957.00 samples/sec Loss 0.8733 Epoch: 21 Global Step: 108600 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:46:37,550-Speed 1972.68 samples/sec Loss 0.8759 Epoch: 21 Global Step: 108650 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:47:03,749-Speed 1954.34 samples/sec Loss 0.8929 Epoch: 21 Global Step: 108700 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:47:30,128-Speed 1940.94 samples/sec Loss 0.9020 Epoch: 21 Global Step: 108750 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:47:56,242-Speed 1960.92 samples/sec Loss 0.8953 Epoch: 21 Global Step: 108800 Fp16 Grad Scale: 16384 Required: 3 hours Training: 2021-03-16 02:48:22,478-Speed 1951.57 samples/sec Loss 0.8748 Epoch: 21 Global Step: 108850 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:48:48,953-Speed 1933.93 samples/sec Loss 0.8851 Epoch: 21 Global Step: 108900 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:49:15,280-Speed 1944.83 samples/sec Loss 0.8813 Epoch: 21 Global Step: 108950 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:49:41,882-Speed 1924.77 samples/sec Loss 0.8938 Epoch: 21 Global Step: 109000 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:50:08,334-Speed 1935.62 samples/sec Loss 0.9026 Epoch: 21 Global Step: 109050 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:50:35,271-Speed 1900.82 samples/sec Loss 0.8991 Epoch: 21 Global Step: 109100 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:51:01,118-Speed 1980.92 samples/sec Loss 0.8949 Epoch: 21 Global Step: 109150 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:51:27,633-Speed 1931.04 samples/sec Loss 0.9144 Epoch: 21 Global Step: 109200 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:51:54,108-Speed 1933.96 samples/sec Loss 0.9008 Epoch: 21 Global Step: 109250 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:52:20,553-Speed 1936.27 samples/sec Loss 0.8834 Epoch: 21 Global Step: 109300 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:52:46,877-Speed 1945.04 samples/sec Loss 0.9012 Epoch: 21 Global Step: 109350 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:53:13,407-Speed 1929.95 samples/sec Loss 0.8827 Epoch: 21 Global Step: 109400 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:53:39,940-Speed 1929.89 samples/sec Loss 0.8997 Epoch: 21 Global Step: 109450 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:54:06,940-Speed 1896.35 samples/sec Loss 0.8955 Epoch: 21 Global Step: 109500 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:54:33,200-Speed 1949.77 samples/sec Loss 0.8924 Epoch: 21 Global Step: 109550 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:54:59,762-Speed 1927.62 samples/sec Loss 0.8952 Epoch: 21 Global Step: 109600 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:55:31,152-Speed 1631.27 samples/sec Loss 0.8248 Epoch: 22 Global Step: 109650 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:55:57,499-Speed 1943.36 samples/sec Loss 0.8453 Epoch: 22 Global Step: 109700 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:56:24,466-Speed 1898.66 samples/sec Loss 0.8201 Epoch: 22 Global Step: 109750 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:56:51,471-Speed 1895.99 samples/sec Loss 0.8243 Epoch: 22 Global Step: 109800 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:57:18,017-Speed 1928.87 samples/sec Loss 0.8698 Epoch: 22 Global Step: 109850 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:57:44,254-Speed 1951.52 samples/sec Loss 0.8444 Epoch: 22 Global Step: 109900 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:58:10,361-Speed 1961.21 samples/sec Loss 0.8602 Epoch: 22 Global Step: 109950 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:58:36,473-Speed 1960.85 samples/sec Loss 0.8292 Epoch: 22 Global Step: 110000 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 02:59:38,146-[lfw][110000]XNorm: 22.395249 Training: 2021-03-16 02:59:38,147-[lfw][110000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 02:59:38,147-[lfw][110000]Accuracy-Highest: 0.99850 Training: 2021-03-16 03:00:49,927-[cfp_fp][110000]XNorm: 21.677314 Training: 2021-03-16 03:00:49,927-[cfp_fp][110000]Accuracy-Flip: 0.99014+-0.00251 Training: 2021-03-16 03:00:49,927-[cfp_fp][110000]Accuracy-Highest: 0.99057 Training: 2021-03-16 03:01:52,471-[agedb_30][110000]XNorm: 22.808573 Training: 2021-03-16 03:01:52,472-[agedb_30][110000]Accuracy-Flip: 0.98483+-0.00709 Training: 2021-03-16 03:01:52,472-[agedb_30][110000]Accuracy-Highest: 0.98483 Training: 2021-03-16 03:02:18,618-Speed 230.48 samples/sec Loss 0.8470 Epoch: 22 Global Step: 110050 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:02:45,002-Speed 1940.72 samples/sec Loss 0.8472 Epoch: 22 Global Step: 110100 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:03:11,533-Speed 1929.97 samples/sec Loss 0.8571 Epoch: 22 Global Step: 110150 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:03:37,709-Speed 1956.04 samples/sec Loss 0.8549 Epoch: 22 Global Step: 110200 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:04:04,063-Speed 1942.85 samples/sec Loss 0.8549 Epoch: 22 Global Step: 110250 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:04:30,273-Speed 1953.51 samples/sec Loss 0.8364 Epoch: 22 Global Step: 110300 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:04:56,869-Speed 1925.16 samples/sec Loss 0.8386 Epoch: 22 Global Step: 110350 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:05:23,390-Speed 1930.63 samples/sec Loss 0.8567 Epoch: 22 Global Step: 110400 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:05:49,492-Speed 1961.60 samples/sec Loss 0.8471 Epoch: 22 Global Step: 110450 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:06:15,785-Speed 1947.32 samples/sec Loss 0.8535 Epoch: 22 Global Step: 110500 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:06:41,627-Speed 1981.39 samples/sec Loss 0.8469 Epoch: 22 Global Step: 110550 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:07:08,025-Speed 1939.55 samples/sec Loss 0.8606 Epoch: 22 Global Step: 110600 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:07:34,385-Speed 1942.47 samples/sec Loss 0.8722 Epoch: 22 Global Step: 110650 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:08:01,298-Speed 1902.52 samples/sec Loss 0.8229 Epoch: 22 Global Step: 110700 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:08:27,596-Speed 1946.98 samples/sec Loss 0.8321 Epoch: 22 Global Step: 110750 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:08:54,089-Speed 1932.58 samples/sec Loss 0.8284 Epoch: 22 Global Step: 110800 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:09:20,573-Speed 1933.33 samples/sec Loss 0.8469 Epoch: 22 Global Step: 110850 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:09:47,153-Speed 1926.31 samples/sec Loss 0.8320 Epoch: 22 Global Step: 110900 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:10:13,158-Speed 1968.88 samples/sec Loss 0.8376 Epoch: 22 Global Step: 110950 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:10:39,469-Speed 1946.07 samples/sec Loss 0.8473 Epoch: 22 Global Step: 111000 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:11:05,871-Speed 1939.24 samples/sec Loss 0.8381 Epoch: 22 Global Step: 111050 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:11:31,885-Speed 1968.23 samples/sec Loss 0.8504 Epoch: 22 Global Step: 111100 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:11:58,311-Speed 1937.61 samples/sec Loss 0.8527 Epoch: 22 Global Step: 111150 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:12:24,549-Speed 1951.40 samples/sec Loss 0.8415 Epoch: 22 Global Step: 111200 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:12:51,125-Speed 1926.63 samples/sec Loss 0.8467 Epoch: 22 Global Step: 111250 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:13:17,612-Speed 1933.04 samples/sec Loss 0.8630 Epoch: 22 Global Step: 111300 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:13:44,122-Speed 1931.41 samples/sec Loss 0.8170 Epoch: 22 Global Step: 111350 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:14:10,674-Speed 1928.40 samples/sec Loss 0.8461 Epoch: 22 Global Step: 111400 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:14:37,475-Speed 1910.41 samples/sec Loss 0.8493 Epoch: 22 Global Step: 111450 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:15:04,352-Speed 1905.07 samples/sec Loss 0.8476 Epoch: 22 Global Step: 111500 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:15:30,401-Speed 1965.55 samples/sec Loss 0.8410 Epoch: 22 Global Step: 111550 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:15:57,132-Speed 1915.47 samples/sec Loss 0.8576 Epoch: 22 Global Step: 111600 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:16:23,411-Speed 1948.41 samples/sec Loss 0.8555 Epoch: 22 Global Step: 111650 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:16:49,400-Speed 1970.11 samples/sec Loss 0.8560 Epoch: 22 Global Step: 111700 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:17:16,081-Speed 1919.06 samples/sec Loss 0.8495 Epoch: 22 Global Step: 111750 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:17:42,238-Speed 1957.42 samples/sec Loss 0.8638 Epoch: 22 Global Step: 111800 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:18:08,581-Speed 1943.69 samples/sec Loss 0.8793 Epoch: 22 Global Step: 111850 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:18:34,643-Speed 1964.57 samples/sec Loss 0.8326 Epoch: 22 Global Step: 111900 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:19:00,803-Speed 1957.26 samples/sec Loss 0.8302 Epoch: 22 Global Step: 111950 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:19:27,413-Speed 1924.20 samples/sec Loss 0.8391 Epoch: 22 Global Step: 112000 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:20:30,449-[lfw][112000]XNorm: 22.246299 Training: 2021-03-16 03:20:30,449-[lfw][112000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 03:20:30,449-[lfw][112000]Accuracy-Highest: 0.99850 Training: 2021-03-16 03:21:44,702-[cfp_fp][112000]XNorm: 21.579025 Training: 2021-03-16 03:21:44,703-[cfp_fp][112000]Accuracy-Flip: 0.99029+-0.00237 Training: 2021-03-16 03:21:44,703-[cfp_fp][112000]Accuracy-Highest: 0.99057 Training: 2021-03-16 03:22:48,482-[agedb_30][112000]XNorm: 22.708033 Training: 2021-03-16 03:22:48,482-[agedb_30][112000]Accuracy-Flip: 0.98467+-0.00682 Training: 2021-03-16 03:22:48,482-[agedb_30][112000]Accuracy-Highest: 0.98483 Training: 2021-03-16 03:23:14,440-Speed 225.53 samples/sec Loss 0.8740 Epoch: 22 Global Step: 112050 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:23:40,504-Speed 1964.44 samples/sec Loss 0.8362 Epoch: 22 Global Step: 112100 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:24:06,845-Speed 1943.83 samples/sec Loss 0.8605 Epoch: 22 Global Step: 112150 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:24:33,135-Speed 1947.55 samples/sec Loss 0.8577 Epoch: 22 Global Step: 112200 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:24:59,158-Speed 1967.55 samples/sec Loss 0.8627 Epoch: 22 Global Step: 112250 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:25:25,847-Speed 1918.43 samples/sec Loss 0.8195 Epoch: 22 Global Step: 112300 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:25:52,789-Speed 1900.46 samples/sec Loss 0.8684 Epoch: 22 Global Step: 112350 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:26:18,983-Speed 1954.70 samples/sec Loss 0.8437 Epoch: 22 Global Step: 112400 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:26:45,667-Speed 1918.79 samples/sec Loss 0.8500 Epoch: 22 Global Step: 112450 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:27:12,332-Speed 1920.22 samples/sec Loss 0.8436 Epoch: 22 Global Step: 112500 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:27:39,207-Speed 1905.17 samples/sec Loss 0.8674 Epoch: 22 Global Step: 112550 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:28:05,654-Speed 1936.00 samples/sec Loss 0.8458 Epoch: 22 Global Step: 112600 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:28:31,941-Speed 1947.77 samples/sec Loss 0.8573 Epoch: 22 Global Step: 112650 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:28:57,449-Speed 2007.23 samples/sec Loss 0.8818 Epoch: 22 Global Step: 112700 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:29:23,663-Speed 1953.23 samples/sec Loss 0.8527 Epoch: 22 Global Step: 112750 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:29:49,986-Speed 1945.32 samples/sec Loss 0.8663 Epoch: 22 Global Step: 112800 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:30:16,342-Speed 1942.66 samples/sec Loss 0.8625 Epoch: 22 Global Step: 112850 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:30:42,926-Speed 1926.20 samples/sec Loss 0.8721 Epoch: 22 Global Step: 112900 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:31:09,732-Speed 1910.04 samples/sec Loss 0.8706 Epoch: 22 Global Step: 112950 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:31:36,194-Speed 1935.05 samples/sec Loss 0.8575 Epoch: 22 Global Step: 113000 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:32:02,771-Speed 1926.48 samples/sec Loss 0.8468 Epoch: 22 Global Step: 113050 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:32:29,370-Speed 1924.94 samples/sec Loss 0.8827 Epoch: 22 Global Step: 113100 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:32:55,621-Speed 1950.49 samples/sec Loss 0.8569 Epoch: 22 Global Step: 113150 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:33:22,106-Speed 1933.27 samples/sec Loss 0.8543 Epoch: 22 Global Step: 113200 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:33:48,875-Speed 1912.68 samples/sec Loss 0.8663 Epoch: 22 Global Step: 113250 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:34:15,464-Speed 1925.64 samples/sec Loss 0.8625 Epoch: 22 Global Step: 113300 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:34:41,565-Speed 1961.70 samples/sec Loss 0.8536 Epoch: 22 Global Step: 113350 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:35:07,841-Speed 1948.59 samples/sec Loss 0.8441 Epoch: 22 Global Step: 113400 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:35:34,731-Speed 1904.13 samples/sec Loss 0.8623 Epoch: 22 Global Step: 113450 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:36:01,061-Speed 1944.59 samples/sec Loss 0.8658 Epoch: 22 Global Step: 113500 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:36:27,284-Speed 1952.55 samples/sec Loss 0.8444 Epoch: 22 Global Step: 113550 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:36:53,667-Speed 1940.76 samples/sec Loss 0.8376 Epoch: 22 Global Step: 113600 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:37:20,010-Speed 1943.69 samples/sec Loss 0.8497 Epoch: 22 Global Step: 113650 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:37:46,432-Speed 1937.85 samples/sec Loss 0.8596 Epoch: 22 Global Step: 113700 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:38:13,036-Speed 1924.55 samples/sec Loss 0.8634 Epoch: 22 Global Step: 113750 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:38:39,572-Speed 1929.56 samples/sec Loss 0.8502 Epoch: 22 Global Step: 113800 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:39:05,498-Speed 1974.87 samples/sec Loss 0.8505 Epoch: 22 Global Step: 113850 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:39:32,054-Speed 1928.25 samples/sec Loss 0.8631 Epoch: 22 Global Step: 113900 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:39:58,602-Speed 1928.66 samples/sec Loss 0.8679 Epoch: 22 Global Step: 113950 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:40:25,021-Speed 1938.06 samples/sec Loss 0.8449 Epoch: 22 Global Step: 114000 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:41:26,665-[lfw][114000]XNorm: 22.390093 Training: 2021-03-16 03:41:26,665-[lfw][114000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 03:41:26,665-[lfw][114000]Accuracy-Highest: 0.99850 Training: 2021-03-16 03:42:40,989-[cfp_fp][114000]XNorm: 21.724905 Training: 2021-03-16 03:42:40,989-[cfp_fp][114000]Accuracy-Flip: 0.98914+-0.00232 Training: 2021-03-16 03:42:40,989-[cfp_fp][114000]Accuracy-Highest: 0.99057 Training: 2021-03-16 03:43:42,504-[agedb_30][114000]XNorm: 22.844678 Training: 2021-03-16 03:43:42,504-[agedb_30][114000]Accuracy-Flip: 0.98483+-0.00709 Training: 2021-03-16 03:43:42,504-[agedb_30][114000]Accuracy-Highest: 0.98483 Training: 2021-03-16 03:44:08,712-Speed 228.89 samples/sec Loss 0.8672 Epoch: 22 Global Step: 114050 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:44:34,803-Speed 1962.43 samples/sec Loss 0.8633 Epoch: 22 Global Step: 114100 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:45:01,592-Speed 1911.23 samples/sec Loss 0.8552 Epoch: 22 Global Step: 114150 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:45:28,271-Speed 1919.22 samples/sec Loss 0.8586 Epoch: 22 Global Step: 114200 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:45:54,333-Speed 1964.74 samples/sec Loss 0.8787 Epoch: 22 Global Step: 114250 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:46:20,425-Speed 1962.48 samples/sec Loss 0.8453 Epoch: 22 Global Step: 114300 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:46:46,482-Speed 1964.98 samples/sec Loss 0.8597 Epoch: 22 Global Step: 114350 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:47:12,839-Speed 1942.59 samples/sec Loss 0.8361 Epoch: 22 Global Step: 114400 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:47:39,343-Speed 1931.87 samples/sec Loss 0.8818 Epoch: 22 Global Step: 114450 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:48:06,155-Speed 1909.65 samples/sec Loss 0.8872 Epoch: 22 Global Step: 114500 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:48:32,634-Speed 1933.69 samples/sec Loss 0.8569 Epoch: 22 Global Step: 114550 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:49:03,724-Speed 1646.87 samples/sec Loss 0.8578 Epoch: 23 Global Step: 114600 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:49:30,152-Speed 1937.41 samples/sec Loss 0.8158 Epoch: 23 Global Step: 114650 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:49:56,769-Speed 1923.62 samples/sec Loss 0.8187 Epoch: 23 Global Step: 114700 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:50:22,729-Speed 1972.39 samples/sec Loss 0.8234 Epoch: 23 Global Step: 114750 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:50:48,971-Speed 1951.14 samples/sec Loss 0.8374 Epoch: 23 Global Step: 114800 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:51:15,486-Speed 1931.02 samples/sec Loss 0.8167 Epoch: 23 Global Step: 114850 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:51:41,781-Speed 1947.27 samples/sec Loss 0.8044 Epoch: 23 Global Step: 114900 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:52:08,118-Speed 1944.16 samples/sec Loss 0.8257 Epoch: 23 Global Step: 114950 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:52:34,197-Speed 1963.27 samples/sec Loss 0.8215 Epoch: 23 Global Step: 115000 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:53:00,328-Speed 1959.45 samples/sec Loss 0.8077 Epoch: 23 Global Step: 115050 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:53:27,232-Speed 1903.12 samples/sec Loss 0.8299 Epoch: 23 Global Step: 115100 Fp16 Grad Scale: 16384 Required: 2 hours Training: 2021-03-16 03:53:53,751-Speed 1930.74 samples/sec Loss 0.8260 Epoch: 23 Global Step: 115150 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:54:20,103-Speed 1943.01 samples/sec Loss 0.8079 Epoch: 23 Global Step: 115200 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:54:46,405-Speed 1946.74 samples/sec Loss 0.8113 Epoch: 23 Global Step: 115250 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:55:12,475-Speed 1964.09 samples/sec Loss 0.8130 Epoch: 23 Global Step: 115300 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:55:39,159-Speed 1918.84 samples/sec Loss 0.8216 Epoch: 23 Global Step: 115350 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:56:06,031-Speed 1905.34 samples/sec Loss 0.8347 Epoch: 23 Global Step: 115400 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:56:32,535-Speed 1931.85 samples/sec Loss 0.8177 Epoch: 23 Global Step: 115450 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:56:58,988-Speed 1935.59 samples/sec Loss 0.7964 Epoch: 23 Global Step: 115500 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:57:25,427-Speed 1936.53 samples/sec Loss 0.8268 Epoch: 23 Global Step: 115550 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:57:51,422-Speed 1969.73 samples/sec Loss 0.8419 Epoch: 23 Global Step: 115600 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:58:17,736-Speed 1945.88 samples/sec Loss 0.8248 Epoch: 23 Global Step: 115650 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:58:44,311-Speed 1926.65 samples/sec Loss 0.8294 Epoch: 23 Global Step: 115700 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:59:10,757-Speed 1936.10 samples/sec Loss 0.8087 Epoch: 23 Global Step: 115750 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 03:59:37,176-Speed 1938.08 samples/sec Loss 0.8234 Epoch: 23 Global Step: 115800 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:00:03,415-Speed 1951.30 samples/sec Loss 0.8119 Epoch: 23 Global Step: 115850 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:00:29,892-Speed 1933.80 samples/sec Loss 0.8015 Epoch: 23 Global Step: 115900 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:00:56,386-Speed 1932.64 samples/sec Loss 0.8290 Epoch: 23 Global Step: 115950 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:01:22,425-Speed 1966.31 samples/sec Loss 0.8354 Epoch: 23 Global Step: 116000 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:02:26,001-[lfw][116000]XNorm: 22.471347 Training: 2021-03-16 04:02:26,002-[lfw][116000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 04:02:26,002-[lfw][116000]Accuracy-Highest: 0.99850 Training: 2021-03-16 04:03:39,249-[cfp_fp][116000]XNorm: 21.750089 Training: 2021-03-16 04:03:39,250-[cfp_fp][116000]Accuracy-Flip: 0.99043+-0.00222 Training: 2021-03-16 04:03:39,250-[cfp_fp][116000]Accuracy-Highest: 0.99057 Training: 2021-03-16 04:04:41,968-[agedb_30][116000]XNorm: 22.896196 Training: 2021-03-16 04:04:41,969-[agedb_30][116000]Accuracy-Flip: 0.98433+-0.00696 Training: 2021-03-16 04:04:41,969-[agedb_30][116000]Accuracy-Highest: 0.98483 Training: 2021-03-16 04:05:08,497-Speed 226.48 samples/sec Loss 0.8306 Epoch: 23 Global Step: 116050 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:05:34,749-Speed 1950.44 samples/sec Loss 0.8022 Epoch: 23 Global Step: 116100 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:06:00,995-Speed 1950.83 samples/sec Loss 0.8339 Epoch: 23 Global Step: 116150 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:06:27,232-Speed 1951.47 samples/sec Loss 0.8133 Epoch: 23 Global Step: 116200 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:06:53,847-Speed 1923.80 samples/sec Loss 0.8206 Epoch: 23 Global Step: 116250 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:07:20,388-Speed 1929.18 samples/sec Loss 0.8232 Epoch: 23 Global Step: 116300 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:07:46,950-Speed 1927.68 samples/sec Loss 0.8081 Epoch: 23 Global Step: 116350 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:08:13,278-Speed 1944.74 samples/sec Loss 0.8132 Epoch: 23 Global Step: 116400 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:08:39,917-Speed 1922.03 samples/sec Loss 0.8093 Epoch: 23 Global Step: 116450 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:09:06,205-Speed 1947.72 samples/sec Loss 0.8001 Epoch: 23 Global Step: 116500 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:09:31,977-Speed 1986.77 samples/sec Loss 0.8387 Epoch: 23 Global Step: 116550 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:09:57,814-Speed 1981.69 samples/sec Loss 0.8382 Epoch: 23 Global Step: 116600 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:10:23,865-Speed 1965.45 samples/sec Loss 0.8314 Epoch: 23 Global Step: 116650 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:10:50,091-Speed 1952.31 samples/sec Loss 0.8326 Epoch: 23 Global Step: 116700 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:11:16,536-Speed 1936.13 samples/sec Loss 0.8253 Epoch: 23 Global Step: 116750 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:11:42,516-Speed 1970.84 samples/sec Loss 0.8284 Epoch: 23 Global Step: 116800 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:12:08,598-Speed 1963.11 samples/sec Loss 0.8107 Epoch: 23 Global Step: 116850 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:12:34,904-Speed 1946.39 samples/sec Loss 0.8344 Epoch: 23 Global Step: 116900 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:13:00,985-Speed 1963.18 samples/sec Loss 0.8156 Epoch: 23 Global Step: 116950 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:13:27,115-Speed 1959.48 samples/sec Loss 0.8286 Epoch: 23 Global Step: 117000 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:13:53,178-Speed 1964.52 samples/sec Loss 0.8239 Epoch: 23 Global Step: 117050 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:14:19,401-Speed 1952.59 samples/sec Loss 0.8278 Epoch: 23 Global Step: 117100 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:14:45,351-Speed 1973.03 samples/sec Loss 0.8236 Epoch: 23 Global Step: 117150 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:15:11,808-Speed 1935.31 samples/sec Loss 0.8182 Epoch: 23 Global Step: 117200 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:15:38,222-Speed 1938.40 samples/sec Loss 0.8381 Epoch: 23 Global Step: 117250 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:16:04,562-Speed 1943.93 samples/sec Loss 0.8231 Epoch: 23 Global Step: 117300 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:16:30,831-Speed 1949.06 samples/sec Loss 0.8246 Epoch: 23 Global Step: 117350 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:16:57,021-Speed 1955.06 samples/sec Loss 0.8316 Epoch: 23 Global Step: 117400 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:17:23,174-Speed 1957.73 samples/sec Loss 0.8416 Epoch: 23 Global Step: 117450 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:17:49,299-Speed 1959.88 samples/sec Loss 0.8479 Epoch: 23 Global Step: 117500 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:18:15,788-Speed 1932.93 samples/sec Loss 0.8315 Epoch: 23 Global Step: 117550 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:18:42,381-Speed 1925.39 samples/sec Loss 0.8350 Epoch: 23 Global Step: 117600 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:19:08,561-Speed 1955.73 samples/sec Loss 0.8259 Epoch: 23 Global Step: 117650 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:19:34,923-Speed 1942.26 samples/sec Loss 0.8374 Epoch: 23 Global Step: 117700 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:20:00,918-Speed 1969.66 samples/sec Loss 0.8260 Epoch: 23 Global Step: 117750 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:20:27,214-Speed 1947.16 samples/sec Loss 0.8357 Epoch: 23 Global Step: 117800 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:20:53,517-Speed 1946.61 samples/sec Loss 0.8106 Epoch: 23 Global Step: 117850 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:21:19,738-Speed 1952.70 samples/sec Loss 0.8210 Epoch: 23 Global Step: 117900 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:21:46,760-Speed 1894.81 samples/sec Loss 0.8322 Epoch: 23 Global Step: 117950 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:22:12,971-Speed 1953.42 samples/sec Loss 0.8127 Epoch: 23 Global Step: 118000 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:23:13,947-[lfw][118000]XNorm: 22.324340 Training: 2021-03-16 04:23:13,948-[lfw][118000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 04:23:13,948-[lfw][118000]Accuracy-Highest: 0.99850 Training: 2021-03-16 04:24:24,338-[cfp_fp][118000]XNorm: 21.624989 Training: 2021-03-16 04:24:24,339-[cfp_fp][118000]Accuracy-Flip: 0.98914+-0.00214 Training: 2021-03-16 04:24:24,339-[cfp_fp][118000]Accuracy-Highest: 0.99057 Training: 2021-03-16 04:25:24,796-[agedb_30][118000]XNorm: 22.766785 Training: 2021-03-16 04:25:24,796-[agedb_30][118000]Accuracy-Flip: 0.98417+-0.00757 Training: 2021-03-16 04:25:24,796-[agedb_30][118000]Accuracy-Highest: 0.98483 Training: 2021-03-16 04:25:50,746-Speed 235.11 samples/sec Loss 0.8370 Epoch: 23 Global Step: 118050 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:26:16,770-Speed 1967.43 samples/sec Loss 0.8298 Epoch: 23 Global Step: 118100 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:26:43,509-Speed 1914.88 samples/sec Loss 0.8305 Epoch: 23 Global Step: 118150 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:27:09,504-Speed 1969.67 samples/sec Loss 0.8388 Epoch: 23 Global Step: 118200 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:27:35,313-Speed 1983.90 samples/sec Loss 0.8293 Epoch: 23 Global Step: 118250 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:28:01,479-Speed 1956.73 samples/sec Loss 0.8147 Epoch: 23 Global Step: 118300 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:28:27,926-Speed 1936.04 samples/sec Loss 0.8301 Epoch: 23 Global Step: 118350 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:28:54,094-Speed 1956.67 samples/sec Loss 0.8234 Epoch: 23 Global Step: 118400 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:29:20,604-Speed 1931.39 samples/sec Loss 0.8348 Epoch: 23 Global Step: 118450 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:29:47,387-Speed 1911.71 samples/sec Loss 0.8414 Epoch: 23 Global Step: 118500 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:30:13,599-Speed 1953.36 samples/sec Loss 0.8273 Epoch: 23 Global Step: 118550 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:30:39,859-Speed 1949.81 samples/sec Loss 0.8280 Epoch: 23 Global Step: 118600 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:31:05,843-Speed 1970.54 samples/sec Loss 0.8383 Epoch: 23 Global Step: 118650 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:31:31,976-Speed 1959.28 samples/sec Loss 0.8236 Epoch: 23 Global Step: 118700 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:31:58,165-Speed 1955.09 samples/sec Loss 0.8253 Epoch: 23 Global Step: 118750 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:32:24,590-Speed 1937.63 samples/sec Loss 0.8542 Epoch: 23 Global Step: 118800 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:32:50,837-Speed 1950.77 samples/sec Loss 0.8342 Epoch: 23 Global Step: 118850 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:33:17,112-Speed 1948.66 samples/sec Loss 0.8294 Epoch: 23 Global Step: 118900 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:33:43,269-Speed 1957.47 samples/sec Loss 0.8397 Epoch: 23 Global Step: 118950 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:34:09,616-Speed 1943.34 samples/sec Loss 0.8456 Epoch: 23 Global Step: 119000 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:34:36,676-Speed 1892.16 samples/sec Loss 0.8317 Epoch: 23 Global Step: 119050 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:35:02,816-Speed 1958.73 samples/sec Loss 0.8306 Epoch: 23 Global Step: 119100 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:35:28,944-Speed 1959.67 samples/sec Loss 0.8196 Epoch: 23 Global Step: 119150 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:35:55,461-Speed 1930.85 samples/sec Loss 0.8350 Epoch: 23 Global Step: 119200 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:36:21,657-Speed 1954.61 samples/sec Loss 0.8328 Epoch: 23 Global Step: 119250 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:36:47,916-Speed 1949.89 samples/sec Loss 0.8362 Epoch: 23 Global Step: 119300 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:37:14,154-Speed 1951.35 samples/sec Loss 0.8230 Epoch: 23 Global Step: 119350 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:37:40,782-Speed 1922.85 samples/sec Loss 0.8060 Epoch: 23 Global Step: 119400 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:38:07,242-Speed 1935.08 samples/sec Loss 0.8462 Epoch: 23 Global Step: 119450 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:38:33,597-Speed 1942.73 samples/sec Loss 0.8195 Epoch: 23 Global Step: 119500 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:39:00,005-Speed 1938.94 samples/sec Loss 0.8377 Epoch: 23 Global Step: 119550 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:39:32,297-Speed 1585.54 samples/sec Loss 0.8040 Epoch: 24 Global Step: 119600 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:39:59,151-Speed 1906.65 samples/sec Loss 0.7848 Epoch: 24 Global Step: 119650 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:40:25,560-Speed 1938.79 samples/sec Loss 0.7873 Epoch: 24 Global Step: 119700 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:40:51,591-Speed 1966.95 samples/sec Loss 0.7983 Epoch: 24 Global Step: 119750 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:41:18,164-Speed 1926.83 samples/sec Loss 0.8002 Epoch: 24 Global Step: 119800 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:41:44,213-Speed 1965.57 samples/sec Loss 0.7951 Epoch: 24 Global Step: 119850 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:42:10,349-Speed 1959.08 samples/sec Loss 0.7730 Epoch: 24 Global Step: 119900 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:42:36,533-Speed 1955.45 samples/sec Loss 0.8185 Epoch: 24 Global Step: 119950 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:43:02,913-Speed 1940.94 samples/sec Loss 0.8212 Epoch: 24 Global Step: 120000 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:44:04,552-[lfw][120000]XNorm: 22.407787 Training: 2021-03-16 04:44:04,553-[lfw][120000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 04:44:04,553-[lfw][120000]Accuracy-Highest: 0.99850 Training: 2021-03-16 04:45:15,777-[cfp_fp][120000]XNorm: 21.731772 Training: 2021-03-16 04:45:15,777-[cfp_fp][120000]Accuracy-Flip: 0.98914+-0.00232 Training: 2021-03-16 04:45:15,777-[cfp_fp][120000]Accuracy-Highest: 0.99057 Training: 2021-03-16 04:46:17,897-[agedb_30][120000]XNorm: 22.867219 Training: 2021-03-16 04:46:17,897-[agedb_30][120000]Accuracy-Flip: 0.98483+-0.00728 Training: 2021-03-16 04:46:17,897-[agedb_30][120000]Accuracy-Highest: 0.98483 Training: 2021-03-16 04:46:44,020-Speed 231.56 samples/sec Loss 0.7916 Epoch: 24 Global Step: 120050 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:47:10,099-Speed 1963.25 samples/sec Loss 0.7807 Epoch: 24 Global Step: 120100 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:47:36,859-Speed 1913.39 samples/sec Loss 0.7853 Epoch: 24 Global Step: 120150 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:48:02,907-Speed 1965.69 samples/sec Loss 0.7910 Epoch: 24 Global Step: 120200 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:48:29,158-Speed 1950.46 samples/sec Loss 0.7811 Epoch: 24 Global Step: 120250 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:48:55,369-Speed 1953.40 samples/sec Loss 0.8012 Epoch: 24 Global Step: 120300 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:49:21,739-Speed 1941.69 samples/sec Loss 0.7898 Epoch: 24 Global Step: 120350 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:49:48,050-Speed 1946.03 samples/sec Loss 0.8096 Epoch: 24 Global Step: 120400 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:50:14,262-Speed 1953.35 samples/sec Loss 0.7655 Epoch: 24 Global Step: 120450 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:50:40,353-Speed 1962.45 samples/sec Loss 0.8020 Epoch: 24 Global Step: 120500 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:51:06,612-Speed 1949.82 samples/sec Loss 0.8143 Epoch: 24 Global Step: 120550 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:51:33,091-Speed 1933.70 samples/sec Loss 0.8062 Epoch: 24 Global Step: 120600 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:51:59,015-Speed 1975.10 samples/sec Loss 0.7970 Epoch: 24 Global Step: 120650 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:52:25,510-Speed 1932.50 samples/sec Loss 0.8055 Epoch: 24 Global Step: 120700 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:52:51,462-Speed 1972.87 samples/sec Loss 0.7809 Epoch: 24 Global Step: 120750 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:53:17,712-Speed 1950.55 samples/sec Loss 0.8029 Epoch: 24 Global Step: 120800 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:53:43,866-Speed 1957.76 samples/sec Loss 0.8146 Epoch: 24 Global Step: 120850 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:54:10,231-Speed 1941.99 samples/sec Loss 0.7901 Epoch: 24 Global Step: 120900 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:54:36,522-Speed 1947.50 samples/sec Loss 0.8074 Epoch: 24 Global Step: 120950 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:55:02,811-Speed 1947.65 samples/sec Loss 0.7819 Epoch: 24 Global Step: 121000 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:55:29,678-Speed 1905.79 samples/sec Loss 0.8041 Epoch: 24 Global Step: 121050 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:55:55,869-Speed 1954.90 samples/sec Loss 0.8041 Epoch: 24 Global Step: 121100 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:56:21,791-Speed 1975.24 samples/sec Loss 0.8077 Epoch: 24 Global Step: 121150 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:56:47,948-Speed 1957.45 samples/sec Loss 0.8006 Epoch: 24 Global Step: 121200 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:57:14,144-Speed 1954.56 samples/sec Loss 0.7891 Epoch: 24 Global Step: 121250 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:57:40,367-Speed 1952.56 samples/sec Loss 0.7936 Epoch: 24 Global Step: 121300 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:58:06,873-Speed 1931.73 samples/sec Loss 0.8030 Epoch: 24 Global Step: 121350 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:58:33,082-Speed 1953.53 samples/sec Loss 0.8234 Epoch: 24 Global Step: 121400 Fp16 Grad Scale: 16384 Required: 1 hours Training: 2021-03-16 04:58:59,164-Speed 1963.10 samples/sec Loss 0.7869 Epoch: 24 Global Step: 121450 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 04:59:25,953-Speed 1911.30 samples/sec Loss 0.8002 Epoch: 24 Global Step: 121500 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 04:59:52,350-Speed 1939.73 samples/sec Loss 0.7883 Epoch: 24 Global Step: 121550 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:00:18,769-Speed 1938.08 samples/sec Loss 0.8037 Epoch: 24 Global Step: 121600 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:00:44,592-Speed 1982.81 samples/sec Loss 0.8055 Epoch: 24 Global Step: 121650 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:01:10,876-Speed 1948.01 samples/sec Loss 0.7960 Epoch: 24 Global Step: 121700 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:01:36,819-Speed 1973.57 samples/sec Loss 0.8140 Epoch: 24 Global Step: 121750 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:02:03,463-Speed 1921.70 samples/sec Loss 0.7894 Epoch: 24 Global Step: 121800 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:02:29,801-Speed 1944.03 samples/sec Loss 0.7885 Epoch: 24 Global Step: 121850 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:02:56,172-Speed 1941.55 samples/sec Loss 0.8165 Epoch: 24 Global Step: 121900 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:03:22,692-Speed 1930.66 samples/sec Loss 0.8144 Epoch: 24 Global Step: 121950 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:03:48,903-Speed 1953.46 samples/sec Loss 0.8043 Epoch: 24 Global Step: 122000 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:04:50,502-[lfw][122000]XNorm: 22.364394 Training: 2021-03-16 05:04:50,502-[lfw][122000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 05:04:50,502-[lfw][122000]Accuracy-Highest: 0.99850 Training: 2021-03-16 05:06:01,022-[cfp_fp][122000]XNorm: 21.678742 Training: 2021-03-16 05:06:01,022-[cfp_fp][122000]Accuracy-Flip: 0.98986+-0.00259 Training: 2021-03-16 05:06:01,022-[cfp_fp][122000]Accuracy-Highest: 0.99057 Training: 2021-03-16 05:07:00,671-[agedb_30][122000]XNorm: 22.811874 Training: 2021-03-16 05:07:00,671-[agedb_30][122000]Accuracy-Flip: 0.98433+-0.00750 Training: 2021-03-16 05:07:00,671-[agedb_30][122000]Accuracy-Highest: 0.98483 Training: 2021-03-16 05:07:26,614-Speed 235.17 samples/sec Loss 0.7970 Epoch: 24 Global Step: 122050 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:07:52,535-Speed 1975.27 samples/sec Loss 0.8208 Epoch: 24 Global Step: 122100 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:08:18,775-Speed 1951.35 samples/sec Loss 0.8010 Epoch: 24 Global Step: 122150 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:08:45,062-Speed 1947.76 samples/sec Loss 0.8048 Epoch: 24 Global Step: 122200 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:09:11,185-Speed 1960.04 samples/sec Loss 0.8058 Epoch: 24 Global Step: 122250 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:09:37,316-Speed 1959.40 samples/sec Loss 0.8024 Epoch: 24 Global Step: 122300 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:10:03,381-Speed 1964.42 samples/sec Loss 0.8087 Epoch: 24 Global Step: 122350 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:10:29,588-Speed 1953.71 samples/sec Loss 0.8054 Epoch: 24 Global Step: 122400 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:10:55,900-Speed 1945.96 samples/sec Loss 0.7999 Epoch: 24 Global Step: 122450 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:11:22,332-Speed 1937.16 samples/sec Loss 0.8366 Epoch: 24 Global Step: 122500 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:11:48,723-Speed 1940.07 samples/sec Loss 0.8105 Epoch: 24 Global Step: 122550 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:12:14,454-Speed 1989.90 samples/sec Loss 0.7724 Epoch: 24 Global Step: 122600 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:12:40,862-Speed 1938.87 samples/sec Loss 0.8103 Epoch: 24 Global Step: 122650 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:13:07,385-Speed 1930.46 samples/sec Loss 0.8033 Epoch: 24 Global Step: 122700 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:13:33,485-Speed 1961.73 samples/sec Loss 0.8094 Epoch: 24 Global Step: 122750 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:13:59,531-Speed 1965.81 samples/sec Loss 0.7979 Epoch: 24 Global Step: 122800 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:14:25,348-Speed 1983.29 samples/sec Loss 0.8017 Epoch: 24 Global Step: 122850 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:14:51,534-Speed 1955.27 samples/sec Loss 0.7841 Epoch: 24 Global Step: 122900 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:15:18,061-Speed 1930.20 samples/sec Loss 0.7986 Epoch: 24 Global Step: 122950 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:15:44,734-Speed 1919.60 samples/sec Loss 0.8346 Epoch: 24 Global Step: 123000 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:16:11,000-Speed 1949.36 samples/sec Loss 0.8085 Epoch: 24 Global Step: 123050 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:16:37,599-Speed 1924.89 samples/sec Loss 0.7931 Epoch: 24 Global Step: 123100 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:17:03,515-Speed 1975.73 samples/sec Loss 0.8073 Epoch: 24 Global Step: 123150 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:17:30,127-Speed 1923.99 samples/sec Loss 0.7855 Epoch: 24 Global Step: 123200 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:17:56,020-Speed 1977.45 samples/sec Loss 0.7905 Epoch: 24 Global Step: 123250 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:18:22,213-Speed 1954.73 samples/sec Loss 0.8155 Epoch: 24 Global Step: 123300 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:18:48,501-Speed 1947.76 samples/sec Loss 0.8214 Epoch: 24 Global Step: 123350 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:19:15,082-Speed 1926.24 samples/sec Loss 0.8098 Epoch: 24 Global Step: 123400 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:19:41,309-Speed 1952.23 samples/sec Loss 0.7912 Epoch: 24 Global Step: 123450 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:20:08,133-Speed 1908.79 samples/sec Loss 0.8087 Epoch: 24 Global Step: 123500 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:20:34,248-Speed 1960.65 samples/sec Loss 0.8182 Epoch: 24 Global Step: 123550 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:21:00,664-Speed 1938.24 samples/sec Loss 0.8063 Epoch: 24 Global Step: 123600 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:21:27,113-Speed 1935.87 samples/sec Loss 0.7894 Epoch: 24 Global Step: 123650 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:21:53,297-Speed 1955.46 samples/sec Loss 0.8111 Epoch: 24 Global Step: 123700 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:22:19,676-Speed 1941.03 samples/sec Loss 0.8179 Epoch: 24 Global Step: 123750 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:22:45,929-Speed 1950.25 samples/sec Loss 0.8092 Epoch: 24 Global Step: 123800 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:23:12,308-Speed 1941.04 samples/sec Loss 0.8164 Epoch: 24 Global Step: 123850 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:23:38,986-Speed 1919.25 samples/sec Loss 0.8183 Epoch: 24 Global Step: 123900 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:24:05,108-Speed 1960.05 samples/sec Loss 0.8163 Epoch: 24 Global Step: 123950 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:24:31,805-Speed 1917.89 samples/sec Loss 0.8149 Epoch: 24 Global Step: 124000 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:25:31,921-[lfw][124000]XNorm: 22.367423 Training: 2021-03-16 05:25:31,921-[lfw][124000]Accuracy-Flip: 0.99833+-0.00211 Training: 2021-03-16 05:25:31,921-[lfw][124000]Accuracy-Highest: 0.99850 Training: 2021-03-16 05:26:42,927-[cfp_fp][124000]XNorm: 21.717782 Training: 2021-03-16 05:26:42,927-[cfp_fp][124000]Accuracy-Flip: 0.98943+-0.00214 Training: 2021-03-16 05:26:42,927-[cfp_fp][124000]Accuracy-Highest: 0.99057 Training: 2021-03-16 05:27:43,230-[agedb_30][124000]XNorm: 22.836980 Training: 2021-03-16 05:27:43,230-[agedb_30][124000]Accuracy-Flip: 0.98367+-0.00756 Training: 2021-03-16 05:27:43,230-[agedb_30][124000]Accuracy-Highest: 0.98483 Training: 2021-03-16 05:28:09,289-Speed 235.42 samples/sec Loss 0.8308 Epoch: 24 Global Step: 124050 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:28:35,391-Speed 1961.62 samples/sec Loss 0.8287 Epoch: 24 Global Step: 124100 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:29:01,417-Speed 1967.31 samples/sec Loss 0.8207 Epoch: 24 Global Step: 124150 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:29:27,629-Speed 1953.35 samples/sec Loss 0.8350 Epoch: 24 Global Step: 124200 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:29:53,676-Speed 1965.72 samples/sec Loss 0.8051 Epoch: 24 Global Step: 124250 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:30:20,575-Speed 1903.50 samples/sec Loss 0.8210 Epoch: 24 Global Step: 124300 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:30:47,106-Speed 1929.87 samples/sec Loss 0.8005 Epoch: 24 Global Step: 124350 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:31:13,474-Speed 1941.79 samples/sec Loss 0.8169 Epoch: 24 Global Step: 124400 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:31:39,891-Speed 1938.22 samples/sec Loss 0.8216 Epoch: 24 Global Step: 124450 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:32:06,393-Speed 1931.96 samples/sec Loss 0.8176 Epoch: 24 Global Step: 124500 Fp16 Grad Scale: 16384 Required: 0 hours Training: 2021-03-16 05:32:33,020-Speed 1922.95 samples/sec Loss 0.8314 Epoch: 24 Global Step: 124550 Fp16 Grad Scale: 16384 Required: -0 hours