BlackboxBench

CIFAR-10 Leaderboard



surrogate model=VGG19-bn
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.985 0.557 0.492 0.512 0.498 0.367 0.105 0.305 0.043 0.429
PGD 0.986 0.621 0.557 0.577 0.554 0.421 0.126 0.35 0.044 0.471
MI-FGSM 0.982 0.6 0.544 0.548 0.52 0.427 0.123 0.316 0.044 0.456
NI-FGSM 0.982 0.598 0.54 0.545 0.519 0.428 0.123 0.317 0.044 0.455
PI-FGSM 0.968 0.738 0.692 0.696 0.673 0.572 0.167 0.426 0.045 0.553
VT 0.983 0.69 0.64 0.652 0.622 0.501 0.148 0.414 0.043 0.521
RAP 0.932 0.704 0.661 0.663 0.649 0.547 0.168 0.439 0.045 0.534
LinBP 1 0.797 0.76 0.79 0.782 0.644 0.248 0.557 0.048 0.625
DI2-FGSM 0.985 0.7 0.656 0.666 0.632 0.529 0.172 0.453 0.044 0.537
SI-FGSM 0.987 0.626 0.572 0.571 0.553 0.443 0.133 0.331 0.045 0.473
Admix 0.986 0.736 0.68 0.696 0.665 0.548 0.182 0.429 0.045 0.552
TI-FGSM 0.985 0.563 0.499 0.516 0.498 0.373 0.108 0.311 0.043 0.433
MI-DI 0.983 0.772 0.764 0.726 0.687 0.64 0.202 0.484 0.049 0.59
MI-DI-TI 0.983 0.771 0.765 0.729 0.684 0.637 0.204 0.485 0.049 0.59
MI-DI-TI-SI 0.989 0.823 0.821 0.781 0.75 0.718 0.274 0.538 0.051 0.638
ILA_BSL 0.972 0.478 0.44 0.435 0.42 0.324 0.09 0.264 0.044 0.385
ILA 0.912 0.72 0.687 0.688 0.668 0.583 0.232 0.47 0.046 0.556
VMI 0.98 0.716 0.677 0.663 0.639 0.552 0.168 0.416 0.046 0.54
VNI 0.98 0.715 0.677 0.662 0.639 0.551 0.171 0.416 0.046 0.54
SI-RAP 0.976 0.757 0.709 0.686 0.692 0.602 0.17 0.446 0.045 0.565
RD 0.899 0.636 0.598 0.623 0.612 0.473 0.154 0.38 0.044 0.491
GhostNet 0.999 0.741 0.697 0.717 0.694 0.535 0.146 0.435 0.044 0.556
LGV 0.968 0.896 0.871 0.895 0.888 0.788 0.366 0.699 0.046 0.713
SWA 0.953 0.852 0.832 0.842 0.835 0.736 0.314 0.611 0.049 0.669
Bayesian_attack 0.984 0.917 0.907 0.914 0.907 0.831 0.438 0.738 0.049 0.743
LGV-GhostNet 0.98 0.904 0.883 0.9 0.897 0.803 0.39 0.725 0.048 0.726
surrogate model=Inception-V3
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.098 0.189 0.283 0.186 0.151 1 0.023 0.126 0.042 0.233
PGD 0.167 0.337 0.476 0.314 0.279 1 0.042 0.187 0.041 0.316
MI-FGSM 0.135 0.229 0.32 0.217 0.171 1 0.031 0.142 0.042 0.254
NI-FGSM 0.135 0.226 0.321 0.215 0.177 1 0.032 0.14 0.043 0.254
PI-FGSM 0.465 0.67 0.807 0.619 0.594 1 0.117 0.364 0.045 0.52
VT 0.284 0.495 0.637 0.455 0.426 0.999 0.071 0.275 0.042 0.409
RAP 0.335 0.542 0.657 0.463 0.475 0.997 0.093 0.289 0.043 0.433
DI2-FGSM 0.266 0.385 0.566 0.363 0.305 1 0.056 0.247 0.045 0.359
SI-FGSM 0.132 0.238 0.362 0.228 0.204 1 0.033 0.141 0.041 0.264
Admix 0.259 0.461 0.603 0.429 0.402 1 0.074 0.242 0.042 0.39
TI-FGSM 0.095 0.184 0.28 0.178 0.15 1 0.024 0.126 0.041 0.231
MI-DI 0.48 0.551 0.754 0.496 0.439 1 0.105 0.328 0.048 0.467
MI-DI-TI 0.479 0.551 0.756 0.497 0.438 1 0.108 0.326 0.048 0.467
MI-DI-TI-SI 0.541 0.615 0.808 0.567 0.5 1 0.148 0.355 0.048 0.509
VMI 0.346 0.527 0.673 0.486 0.435 1 0.093 0.287 0.042 0.432
VNI-FGSM 0.351 0.531 0.677 0.489 0.43 1 0.093 0.284 0.044 0.433
SI-RAP 0.318 0.521 0.651 0.433 0.454 0.999 0.089 0.258 0.041 0.418
GhostNet 0.573 0.764 0.85 0.738 0.759 0.996 0.227 0.491 0.044 0.605
ILA_BSL 0.108 0.199 0.302 0.179 0.171 1 0.026 0.126 0.04 0.239
ILA 0.334 0.498 0.6 0.46 0.441 0.959 0.12 0.285 0.042 0.415
RD 0.367 0.647 0.81 0.682 0.605 1 0.107 0.358 0.044 0.513
LGV 0.788 0.943 0.964 0.951 0.948 0.993 0.453 0.727 0.047 0.757
SWA 0.75 0.938 0.969 0.94 0.936 1 0.443 0.722 0.049 0.75
Bayesian_attack 0.809 0.961 0.982 0.962 0.96 1 0.52 0.774 0.05 0.78
LGV-GhostNet 0.763 0.884 0.916 0.89 0.891 0.963 0.446 0.693 0.049 0.722
surrogate model=ResNet-50
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.142 0.299 1 0.379 0.234 0.216 0.049 0.234 0.042 0.288
PGD 0.241 0.5 1 0.597 0.415 0.371 0.079 0.365 0.042 0.401
MI-FGSM 0.22 0.386 1 0.454 0.293 0.293 0.073 0.258 0.043 0.336
NI-FGSM 0.22 0.387 1 0.452 0.292 0.29 0.071 0.267 0.043 0.336
PI-FGSM 0.53 0.766 1 0.816 0.678 0.644 0.164 0.538 0.045 0.576
VT 0.422 0.689 1 0.766 0.608 0.568 0.135 0.526 0.043 0.528
RAP 0.564 0.817 0.998 0.842 0.754 0.718 0.202 0.639 0.044 0.62
DI2-FGSM 0.399 0.601 1 0.668 0.492 0.514 0.125 0.478 0.045 0.48
SI-FGSM 0.197 0.389 1 0.475 0.309 0.312 0.068 0.273 0.042 0.341
Admix 0.389 0.66 1 0.763 0.573 0.538 0.148 0.467 0.042 0.509
TI-FGSM 0.141 0.301 1 0.376 0.236 0.218 0.051 0.236 0.041 0.289
MI-DI 0.662 0.779 1 0.817 0.66 0.726 0.198 0.596 0.05 0.61
MI-DI-TI 0.661 0.782 1 0.812 0.652 0.722 0.197 0.598 0.05 0.608
MI-DI-TI-SI 0.729 0.825 1 0.866 0.736 0.798 0.279 0.635 0.049 0.657
VMI 0.501 0.727 1 0.797 0.616 0.608 0.162 0.517 0.045 0.553
VNI 0.494 0.719 1 0.792 0.618 0.608 0.16 0.519 0.044 0.55
SI-RAP 0.592 0.825 0.999 0.849 0.76 0.753 0.215 0.615 0.044 0.628
SGM 0.636 0.85 1 0.921 0.848 0.747 0.313 0.74 0.051 0.678
GhostNet 0.702 0.921 1 0.956 0.917 0.864 0.378 0.809 0.048 0.733
ILA_BSL 0.152 0.307 1 0.391 0.257 0.245 0.051 0.253 0.04 0.3
ILA 0.354 0.537 0.994 0.589 0.449 0.453 0.145 0.396 0.042 0.44
LinBP 0.655 0.896 1 0.935 0.893 0.811 0.334 0.751 0.048 0.703
RD 0.483 0.768 1 0.858 0.725 0.647 0.223 0.599 0.045 0.594
LGV 0.876 0.974 1 0.987 0.975 0.953 0.619 0.908 0.05 0.816
SWA 0.823 0.966 1 0.983 0.965 0.934 0.571 0.88 0.051 0.797
Bayesian_attack 0.876 0.98 1 0.989 0.979 0.96 0.644 0.913 0.051 0.821
LGV-GhostNet 0.859 0.952 0.997 0.973 0.952 0.933 0.609 0.894 0.051 0.802
surrogate model=DenseNet-121
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.094 0.323 0.195 0.548 1 0.088 0.08 0.346 0.041 0.302
PGD 0.155 0.562 0.336 0.791 1 0.146 0.176 0.551 0.043 0.418
MI-FGSM 0.13 0.406 0.264 0.634 1 0.131 0.093 0.395 0.043 0.344
NI-FGSM 0.129 0.41 0.263 0.644 1 0.131 0.092 0.392 0.042 0.345
PI-FGSM 0.285 0.738 0.492 0.9 1 0.238 0.243 0.624 0.043 0.507
VT 0.278 0.746 0.501 0.888 1 0.256 0.272 0.704 0.042 0.521
RAP 0.324 0.786 0.547 0.878 0.992 0.318 0.35 0.71 0.044 0.55
DI2-FGSM 0.25 0.65 0.448 0.843 1 0.229 0.234 0.651 0.044 0.483
SI-FGSM 0.124 0.432 0.279 0.663 1 0.128 0.113 0.398 0.042 0.353
Admix 0.184 0.602 0.387 0.836 1 0.172 0.221 0.547 0.042 0.443
TI-FGSM 0.093 0.327 0.2 0.552 1 0.091 0.078 0.354 0.041 0.304
MI-DI 0.426 0.8 0.606 0.915 1 0.345 0.278 0.714 0.047 0.57
MI-DI-TI 0.423 0.8 0.605 0.918 1 0.353 0.276 0.717 0.047 0.571
MI-DI-TI-SI 0.499 0.855 0.685 0.95 1 0.439 0.39 0.753 0.047 0.624
VMI 0.339 0.8 0.572 0.93 1 0.3 0.289 0.708 0.044 0.554
VNI 0.342 0.799 0.573 0.928 1 0.303 0.293 0.707 0.044 0.554
SI-RAP 0.372 0.84 0.611 0.916 0.998 0.374 0.41 0.731 0.042 0.588
SGM 0.331 0.737 0.57 0.912 1 0.36 0.33 0.762 0.047 0.561
GhostNet 0.285 0.778 0.536 0.908 1 0.34 0.36 0.71 0.043 0.551
ILA_BSL 0.106 0.368 0.223 0.585 1 0.108 0.112 0.392 0.041 0.326
ILA 0.218 0.461 0.37 0.556 0.905 0.196 0.222 0.456 0.042 0.381
RD 0.121 0.477 0.295 0.77 1 0.128 0.139 0.491 0.042 0.385
LGV 0.799 0.979 0.932 0.994 0.999 0.844 0.805 0.946 0.049 0.816
SWA 0.636 0.947 0.846 0.982 0.999 0.713 0.782 0.919 0.047 0.764
Bayesian_attack 0.781 0.982 0.932 0.995 1 0.848 0.884 0.961 0.049 0.826
LGV-GhostNet 0.791 0.966 0.914 0.986 0.998 0.835 0.778 0.937 0.049 0.806
surrogate model=VGG19-bn
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.986 0.509 0.43 0.477 0.468 0.334 0.152 0.298 0.042 0.411
PGD 0.983 0.583 0.507 0.543 0.525 0.4 0.177 0.331 0.042 0.455
MI-FGSM 0.974 0.524 0.464 0.481 0.471 0.378 0.154 0.3 0.042 0.421
NI-FGSM 0.974 0.527 0.462 0.483 0.47 0.376 0.154 0.298 0.042 0.421
PI-FGSM 0.961 0.68 0.621 0.65 0.644 0.525 0.223 0.436 0.044 0.531
VT 0.985 0.553 0.478 0.526 0.514 0.375 0.183 0.332 0.042 0.443
LinBP 1 0.746 0.682 0.726 0.732 0.597 0.346 0.513 0.044 0.599
RAP 0.956 0.632 0.568 0.601 0.604 0.478 0.203 0.381 0.043 0.496
DI2-FGSM 0.986 0.682 0.634 0.649 0.632 0.525 0.251 0.438 0.045 0.538
SI-FGSM 0.986 0.56 0.491 0.513 0.513 0.392 0.167 0.306 0.043 0.441
Admix 0.985 0.662 0.581 0.614 0.614 0.478 0.207 0.375 0.043 0.507
TI-FGSM 0.986 0.51 0.431 0.479 0.468 0.337 0.157 0.298 0.042 0.412
MI-DI 0.977 0.733 0.709 0.697 0.666 0.605 0.263 0.489 0.048 0.576
MI-DI-TI 0.976 0.731 0.708 0.697 0.669 0.601 0.26 0.49 0.047 0.575
MI-DI-TI-SI 0.984 0.765 0.748 0.724 0.712 0.663 0.292 0.501 0.048 0.604
ILA_BSL 0.968 0.486 0.439 0.443 0.43 0.349 0.128 0.271 0.042 0.395
ILA 0.912 0.667 0.617 0.641 0.63 0.535 0.286 0.447 0.043 0.531
VMI 0.974 0.57 0.51 0.533 0.52 0.417 0.183 0.34 0.042 0.454
VNI 0.974 0.57 0.509 0.534 0.521 0.418 0.183 0.341 0.042 0.455
SI-RAP 0.982 0.659 0.587 0.598 0.609 0.509 0.189 0.363 0.043 0.504
RD 0.895 0.607 0.562 0.606 0.603 0.455 0.234 0.387 0.042 0.488
GhostNet 1 0.755 0.677 0.724 0.721 0.559 0.25 0.466 0.045 0.577
LGV 0.965 0.879 0.836 0.862 0.868 0.773 0.484 0.663 0.045 0.708
SWA 0.937 0.805 0.773 0.783 0.79 0.694 0.405 0.552 0.046 0.643
Bayesian_attack 0.974 0.884 0.861 0.863 0.874 0.801 0.514 0.668 0.048 0.721
LGV-GhostNet 0.98 0.903 0.867 0.887 0.896 0.817 0.527 0.701 0.046 0.736
surrogate model=Inception-V3
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.075 0.159 0.224 0.177 0.145 1 0.042 0.126 0.04 0.221
PGD 0.155 0.329 0.435 0.332 0.295 1 0.074 0.201 0.041 0.318
MI-FGSM 0.112 0.211 0.285 0.211 0.192 1 0.054 0.146 0.041 0.25
NI-FGSM 0.111 0.211 0.284 0.209 0.189 1 0.052 0.146 0.041 0.249
PI-FGSM 0.443 0.669 0.743 0.634 0.635 1 0.202 0.413 0.042 0.531
VT 0.099 0.217 0.312 0.236 0.205 1 0.058 0.157 0.04 0.258
RAP 0.238 0.444 0.522 0.395 0.409 0.993 0.105 0.239 0.04 0.376
DI2-FGSM 0.276 0.425 0.587 0.424 0.365 1 0.109 0.285 0.043 0.39
SI-FGSM 0.095 0.198 0.295 0.215 0.188 1 0.051 0.138 0.04 0.247
Admix 0.219 0.441 0.542 0.429 0.403 1 0.105 0.247 0.04 0.381
TI-FGSM 0.071 0.162 0.227 0.171 0.147 1 0.041 0.123 0.04 0.22
MI-DI 0.432 0.557 0.718 0.526 0.48 1 0.147 0.352 0.045 0.473
MI-DI-TI 0.431 0.553 0.714 0.528 0.478 1 0.15 0.352 0.045 0.472
MI-DI-TI-SI 0.473 0.605 0.751 0.572 0.527 1 0.174 0.36 0.045 0.501
VMI 0.143 0.277 0.371 0.278 0.249 1 0.073 0.188 0.041 0.291
VNI 0.143 0.276 0.372 0.278 0.247 1 0.075 0.186 0.041 0.291
SI-RAP 0.225 0.427 0.53 0.38 0.391 0.997 0.094 0.221 0.04 0.367
GhostNet 0.685 0.863 0.909 0.851 0.869 0.999 0.435 0.62 0.046 0.697
ILA_BSL 0.126 0.233 0.341 0.228 0.215 1 0.043 0.14 0.039 0.263
ILA 0.275 0.451 0.538 0.422 0.406 0.978 0.146 0.274 0.042 0.392
RD 0.311 0.624 0.768 0.68 0.604 1 0.224 0.415 0.043 0.519
LGV 0.739 0.933 0.95 0.944 0.943 0.992 0.64 0.763 0.046 0.772
SWA 0.631 0.905 0.937 0.913 0.909 0.999 0.593 0.714 0.045 0.738
Bayesian_attack 0.7 0.936 0.96 0.943 0.94 1 0.651 0.77 0.046 0.772
LGV-GhostNet 0.817 0.924 0.937 0.926 0.935 0.979 0.644 0.762 0.05 0.775
surrogate model=ResNet-50
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.102 0.234 1 0.336 0.203 0.171 0.071 0.234 0.04 0.266
PGD 0.207 0.455 1 0.582 0.393 0.338 0.116 0.382 0.041 0.391
MI-FGSM 0.164 0.325 1 0.423 0.282 0.255 0.105 0.296 0.041 0.321
NI-FGSM 0.163 0.326 1 0.421 0.277 0.256 0.103 0.295 0.041 0.32
PI-FGSM 0.441 0.726 1 0.795 0.673 0.607 0.257 0.615 0.042 0.573
VT 0.136 0.32 1 0.44 0.281 0.223 0.097 0.296 0.04 0.315
RAP 0.375 0.685 0.995 0.752 0.64 0.564 0.222 0.552 0.041 0.536
DI2-FGSM 0.371 0.574 1 0.673 0.494 0.5 0.2 0.51 0.045 0.485
SI-FGSM 0.141 0.318 1 0.439 0.275 0.246 0.088 0.273 0.04 0.313
Admix 0.309 0.606 1 0.734 0.536 0.481 0.182 0.481 0.041 0.485
TI-FGSM 0.102 0.234 1 0.339 0.203 0.171 0.071 0.234 0.04 0.266
MI-DI 0.532 0.733 1 0.802 0.657 0.67 0.283 0.652 0.045 0.597
MI-DI-TI 0.531 0.731 1 0.802 0.66 0.661 0.284 0.648 0.046 0.596
MI-DI-TI-SI 0.599 0.777 1 0.844 0.717 0.747 0.325 0.668 0.045 0.636
VMI 0.203 0.405 1 0.529 0.356 0.311 0.137 0.369 0.041 0.372
VNI 0.201 0.405 1 0.533 0.356 0.313 0.138 0.369 0.041 0.373
SI-RAP 0.39 0.705 0.999 0.767 0.653 0.606 0.214 0.53 0.04 0.545
SGM 0.456 0.739 1 0.851 0.758 0.625 0.439 0.694 0.048 0.623
GhostNet 0.664 0.917 1 0.96 0.921 0.868 0.602 0.85 0.046 0.759
ILA_BSL 0.165 0.34 1 0.455 0.294 0.288 0.081 0.298 0.04 0.329
ILA 0.265 0.446 0.998 0.531 0.395 0.384 0.155 0.377 0.041 0.399
LinBP 0.518 0.838 1 0.896 0.843 0.727 0.493 0.735 0.045 0.677
RD 0.376 0.705 1 0.828 0.688 0.583 0.366 0.625 0.043 0.579
LGV 0.815 0.967 0.999 0.981 0.966 0.942 0.736 0.905 0.048 0.818
SWA 0.722 0.943 1 0.969 0.944 0.893 0.688 0.857 0.046 0.785
Bayesian_attack 0.774 0.963 1 0.981 0.962 0.931 0.74 0.893 0.047 0.81
LGV-GhostNet 0.843 0.952 0.998 0.972 0.957 0.935 0.742 0.9 0.05 0.817
surrogate model=DenseNet-121
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.068 0.266 0.156 0.484 1 0.08 0.166 0.355 0.04 0.29
PGD 0.136 0.538 0.309 0.785 1 0.147 0.319 0.569 0.041 0.427
MI-FGSM 0.093 0.335 0.206 0.564 1 0.106 0.197 0.414 0.041 0.328
NI-FGSM 0.093 0.338 0.207 0.564 1 0.105 0.198 0.414 0.041 0.329
PI-FGSM 0.231 0.696 0.437 0.868 1 0.244 0.457 0.681 0.04 0.517
VT 0.089 0.369 0.218 0.632 1 0.106 0.232 0.451 0.041 0.349
RAP 0.074 0.314 0.181 0.561 0.999 0.087 0.196 0.392 0.041 0.316
DI2-FGSM 0.233 0.648 0.442 0.844 1 0.234 0.417 0.676 0.042 0.504
SI-FGSM 0.084 0.358 0.219 0.606 1 0.105 0.209 0.4 0.04 0.336
Admix 0.131 0.534 0.315 0.8 1 0.141 0.331 0.548 0.041 0.427
TI-FGSM 0.068 0.262 0.158 0.485 1 0.079 0.166 0.353 0.041 0.29
MI-DI 0.296 0.729 0.518 0.885 1 0.291 0.48 0.737 0.043 0.553
MI-DI-TI 0.297 0.728 0.516 0.883 1 0.289 0.478 0.735 0.043 0.552
MI-DI-TI-SI 0.372 0.791 0.603 0.921 1 0.379 0.553 0.766 0.043 0.603
VMI 0.114 0.443 0.267 0.698 1 0.135 0.281 0.515 0.042 0.388
VNI 0.115 0.442 0.27 0.701 1 0.135 0.282 0.514 0.041 0.389
SI-RAP 0.096 0.417 0.247 0.683 0.999 0.118 0.253 0.438 0.04 0.366
SGM 0.314 0.704 0.53 0.877 1 0.381 0.524 0.747 0.044 0.569
GhostNet 0.325 0.819 0.596 0.932 1 0.41 0.649 0.787 0.042 0.618
ILA_BSL 0.115 0.419 0.258 0.631 1 0.139 0.205 0.438 0.04 0.361
ILA 0.169 0.444 0.33 0.566 0.932 0.18 0.275 0.467 0.04 0.378
RD 0.1 0.428 0.254 0.738 1 0.116 0.313 0.53 0.041 0.391
LGV 0.747 0.971 0.91 0.991 0.999 0.819 0.872 0.937 0.047 0.81
SWA 0.592 0.949 0.835 0.983 1 0.715 0.881 0.92 0.046 0.769
Bayesian_attack 0.709 0.971 0.897 0.992 1 0.811 0.911 0.95 0.046 0.81
LGV-GhostNet 0.769 0.963 0.905 0.984 0.997 0.835 0.85 0.925 0.048 0.808
surrogate model=VGG19-bn
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 1 0.693 0.675 0.703 0.677 0.598 0.279 0.525 0.325 0.608
PGD 0.999 0.766 0.74 0.767 0.735 0.667 0.317 0.587 0.326 0.656
MI-FGSM 0.998 0.711 0.693 0.711 0.689 0.615 0.287 0.541 0.326 0.619
NI-FGSM 0.998 0.712 0.693 0.715 0.69 0.615 0.287 0.542 0.326 0.62
PI-FGSM 0.999 0.81 0.783 0.803 0.785 0.705 0.345 0.626 0.327 0.687
VT 1 0.732 0.711 0.739 0.711 0.636 0.307 0.558 0.325 0.635
RAP 0.998 0.797 0.774 0.796 0.781 0.691 0.357 0.622 0.327 0.683
LinBP 1 0.926 0.913 0.928 0.916 0.851 0.569 0.778 0.328 0.801
DI2-FGSM 1 0.84 0.835 0.834 0.808 0.766 0.385 0.662 0.333 0.718
SI-FGSM 1 0.735 0.722 0.742 0.721 0.658 0.294 0.545 0.326 0.638
Admix 1 0.817 0.788 0.813 0.791 0.721 0.345 0.612 0.327 0.69
TI-FGSM 1 0.693 0.679 0.705 0.676 0.595 0.283 0.53 0.326 0.609
MI-DI 0.998 0.829 0.824 0.828 0.801 0.755 0.373 0.651 0.333 0.71
MI-DI-TI 0.998 0.828 0.822 0.832 0.8 0.757 0.373 0.652 0.334 0.711
MI-DI-TI-SI 0.999 0.872 0.883 0.871 0.851 0.831 0.419 0.684 0.338 0.75
FIA 0.999 0.836 0.821 0.851 0.839 0.765 0.424 0.715 0.333 0.731
NAA 0.999 0.894 0.872 0.891 0.896 0.833 0.532 0.738 0.329 0.776
ILA_BSL 0.981 0.529 0.544 0.522 0.504 0.468 0.135 0.332 0.319 0.481
ILA 0.989 0.885 0.874 0.885 0.871 0.82 0.478 0.74 0.327 0.763
RD 0.985 0.817 0.794 0.824 0.807 0.726 0.399 0.639 0.323 0.702
VMI 0.998 0.749 0.727 0.746 0.722 0.659 0.313 0.568 0.325 0.645
VNI 0.998 0.748 0.728 0.745 0.722 0.658 0.315 0.569 0.325 0.645
SI-RAP 0.988 0.843 0.832 0.829 0.832 0.781 0.365 0.623 0.331 0.714
GhostNet 1 0.882 0.868 0.879 0.863 0.798 0.416 0.691 0.329 0.747
LGV 1 0.973 0.965 0.972 0.965 0.938 0.649 0.859 0.331 0.85
SWA 0.999 0.938 0.934 0.932 0.927 0.889 0.588 0.768 0.331 0.812
Bayesian_attack 1 0.972 0.971 0.966 0.964 0.94 0.678 0.846 0.334 0.852
LGV-GhostNet 1 0.977 0.971 0.977 0.973 0.952 0.672 0.876 0.335 0.859
surrogate model=Inception-V3
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.423 0.55 0.678 0.586 0.536 0.998 0.191 0.423 0.321 0.523
PGD 0.544 0.682 0.797 0.713 0.667 1 0.246 0.511 0.32 0.609
MI-FGSM 0.429 0.561 0.691 0.593 0.556 0.997 0.189 0.43 0.321 0.53
NI-FGSM 0.431 0.562 0.694 0.596 0.554 0.997 0.188 0.43 0.321 0.53
PI-FGSM 0.656 0.792 0.874 0.801 0.779 0.998 0.325 0.602 0.323 0.683
VT 0.481 0.614 0.74 0.647 0.603 0.999 0.221 0.468 0.324 0.566
RAP 0.626 0.765 0.851 0.782 0.756 0.952 0.287 0.59 0.324 0.659
DI2-FGSM 0.698 0.785 0.892 0.792 0.754 0.999 0.319 0.623 0.332 0.688
SI-FGSM 0.455 0.578 0.713 0.615 0.577 0.999 0.202 0.433 0.323 0.544
Admix 0.589 0.728 0.837 0.757 0.713 1 0.283 0.541 0.322 0.641
TI-FGSM 0.42 0.545 0.679 0.58 0.533 0.999 0.197 0.428 0.321 0.522
MI-DI 0.703 0.791 0.888 0.79 0.759 0.996 0.305 0.616 0.33 0.687
MI-DI-TI 0.699 0.786 0.89 0.79 0.758 0.996 0.307 0.616 0.33 0.686
MI-DI-TI-SI 0.778 0.85 0.926 0.842 0.816 0.998 0.367 0.654 0.335 0.729
VMI 0.491 0.632 0.759 0.663 0.622 0.997 0.226 0.478 0.322 0.576
VNI 0.49 0.631 0.759 0.663 0.622 0.997 0.224 0.476 0.322 0.576
SI-RAP 0.838 0.895 0.929 0.892 0.896 0.99 0.479 0.692 0.333 0.772
GhostNet 0.882 0.936 0.974 0.94 0.941 1 0.537 0.787 0.336 0.815
FIA 0.815 0.908 0.936 0.906 0.913 0.979 0.558 0.802 0.337 0.795
NAA 0.786 0.866 0.901 0.859 0.877 0.986 0.547 0.72 0.336 0.764
ILA_BSL 0.366 0.487 0.603 0.481 0.47 0.911 0.117 0.305 0.315 0.45
ILA 0.75 0.831 0.859 0.819 0.827 0.941 0.498 0.702 0.329 0.729
RD 0.725 0.872 0.955 0.911 0.867 1 0.439 0.732 0.326 0.758
LGV 0.962 0.993 0.996 0.996 0.993 1 0.83 0.942 0.336 0.894
SWA 0.923 0.989 0.996 0.992 0.988 1 0.793 0.92 0.33 0.881
Bayesian_attack 0.947 0.993 0.999 0.995 0.993 1 0.833 0.942 0.332 0.893
LGV-GhostNet 0.948 0.971 0.988 0.978 0.976 0.997 0.673 0.888 0.341 0.862
surrogate model=ResNet-50
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.585 0.767 1 0.867 0.747 0.741 0.349 0.725 0.326 0.679
PGD 0.714 0.87 1 0.935 0.845 0.845 0.425 0.809 0.327 0.752
MI-FGSM 0.603 0.791 1 0.887 0.766 0.763 0.368 0.748 0.325 0.694
NI-FGSM 0.603 0.789 1 0.883 0.766 0.763 0.367 0.744 0.325 0.693
PI-FGSM 0.774 0.909 1 0.952 0.89 0.875 0.49 0.847 0.326 0.785
VT 0.639 0.815 1 0.899 0.793 0.786 0.402 0.765 0.326 0.714
RAP 0.781 0.917 1 0.953 0.897 0.896 0.492 0.845 0.326 0.789
DI2-FGSM 0.823 0.922 1 0.956 0.895 0.909 0.535 0.875 0.334 0.806
SI-FGSM 0.632 0.806 1 0.898 0.781 0.793 0.38 0.741 0.323 0.706
Admix 0.762 0.907 1 0.956 0.884 0.887 0.493 0.83 0.325 0.783
TI-FGSM 0.579 0.767 1 0.865 0.739 0.738 0.349 0.727 0.326 0.677
MI-DI 0.829 0.929 1 0.96 0.901 0.915 0.534 0.884 0.335 0.81
MI-DI-TI 0.827 0.93 1 0.961 0.901 0.914 0.538 0.885 0.336 0.81
MI-DI-TI-SI 0.888 0.953 1 0.973 0.935 0.952 0.606 0.899 0.336 0.838
VMI 0.662 0.841 1 0.917 0.817 0.818 0.421 0.789 0.325 0.732
VNI 0.665 0.842 1 0.918 0.818 0.815 0.42 0.79 0.325 0.733
SI-RAP 0.889 0.95 1 0.963 0.945 0.953 0.569 0.83 0.331 0.826
SGM 0.778 0.916 1 0.969 0.914 0.886 0.663 0.903 0.33 0.818
GhostNet 0.891 0.971 1 0.989 0.968 0.963 0.687 0.944 0.335 0.861
FIA 0.787 0.878 0.982 0.896 0.875 0.877 0.526 0.842 0.34 0.778
NAA 0.689 0.783 0.94 0.797 0.785 0.787 0.501 0.68 0.336 0.7
ILA_BSL 0.499 0.682 0.99 0.762 0.655 0.686 0.217 0.564 0.317 0.597
ILA 0.83 0.915 0.997 0.938 0.908 0.908 0.6 0.851 0.324 0.808
LinBP 0.843 0.965 1 0.987 0.965 0.948 0.744 0.927 0.325 0.856
RD 0.779 0.929 1 0.974 0.913 0.916 0.561 0.883 0.33 0.809
LGV 0.981 0.998 1 0.999 0.997 0.997 0.896 0.986 0.333 0.91
SWA 0.951 0.996 1 0.999 0.994 0.991 0.882 0.978 0.332 0.903
Bayesian_attack 0.967 0.998 1 0.999 0.997 0.995 0.908 0.986 0.333 0.909
LGV-GhostNet 0.973 0.991 1 0.996 0.989 0.991 0.818 0.974 0.341 0.897
surrogate model=DenseNet-121
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.433 0.75 0.64 0.921 1 0.497 0.528 0.818 0.323 0.657
PGD 0.574 0.889 0.764 0.978 1 0.616 0.628 0.901 0.323 0.742
MI-FGSM 0.473 0.797 0.677 0.945 1 0.523 0.558 0.85 0.323 0.683
NI-FGSM 0.469 0.797 0.678 0.946 1 0.524 0.555 0.849 0.323 0.682
PI-FGSM 0.594 0.904 0.782 0.98 1 0.621 0.663 0.911 0.322 0.753
VT 0.496 0.819 0.704 0.952 1 0.549 0.589 0.861 0.322 0.699
RAP 0.55 0.886 0.754 0.975 1 0.595 0.628 0.892 0.324 0.734
DI2-FGSM 0.71 0.931 0.863 0.986 1 0.725 0.735 0.938 0.33 0.802
SI-FGSM 0.473 0.79 0.678 0.94 1 0.536 0.556 0.823 0.322 0.68
Admix 0.551 0.87 0.753 0.976 1 0.597 0.643 0.88 0.323 0.733
TI-FGSM 0.435 0.746 0.638 0.922 1 0.494 0.524 0.819 0.323 0.655
MI-DI 0.715 0.939 0.864 0.987 1 0.731 0.734 0.942 0.329 0.804
MI-DI-TI 0.715 0.938 0.866 0.987 1 0.728 0.737 0.942 0.329 0.805
MI-DI-TI-SI 0.772 0.96 0.906 0.991 1 0.797 0.794 0.947 0.332 0.833
VMI 0.528 0.851 0.74 0.965 1 0.576 0.618 0.883 0.324 0.721
VNI 0.526 0.851 0.74 0.966 1 0.576 0.617 0.883 0.324 0.72
SI-RAP 0.699 0.939 0.833 0.977 1 0.746 0.701 0.871 0.323 0.788
SGM 0.585 0.894 0.807 0.981 1 0.663 0.771 0.932 0.328 0.773
GhostNet 0.633 0.935 0.84 0.99 1 0.709 0.807 0.945 0.324 0.798
FIA 0.607 0.831 0.727 0.886 0.992 0.658 0.582 0.846 0.332 0.718
NAA 0.682 0.874 0.787 0.903 0.977 0.753 0.705 0.824 0.333 0.76
ILA_BSL 0.35 0.674 0.537 0.802 0.996 0.419 0.346 0.641 0.312 0.564
ILA 0.67 0.894 0.814 0.949 0.994 0.708 0.674 0.877 0.319 0.767
RD 0.523 0.851 0.748 0.972 1 0.584 0.651 0.898 0.324 0.728
LGV 0.97 0.999 0.997 1 1 0.985 0.972 0.996 0.337 0.917
SWA 0.928 0.998 0.991 1 1 0.967 0.983 0.996 0.334 0.911
Bayesian_attack 0.964 0.999 0.996 1 1 0.983 0.989 0.998 0.338 0.919
LGV-GhostNet 0.963 0.997 0.992 1 1 0.974 0.928 0.992 0.338 0.909
surrogate model=VGG19-bn
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 1 0.721 0.711 0.722 0.695 0.612 0.245 0.575 0.326 0.623
PGD 1 0.788 0.77 0.794 0.756 0.678 0.28 0.646 0.329 0.671
MI-FGSM 0.999 0.779 0.772 0.782 0.753 0.687 0.277 0.634 0.334 0.669
NI-FGSM 0.999 0.781 0.767 0.781 0.752 0.686 0.28 0.637 0.332 0.668
PI-FGSM 0.999 0.856 0.838 0.85 0.826 0.757 0.331 0.701 0.334 0.721
VT 0.999 0.83 0.82 0.83 0.795 0.733 0.303 0.678 0.33 0.702
RAP 0.972 0.861 0.846 0.855 0.839 0.775 0.37 0.738 0.334 0.732
LinBP 1 0.941 0.934 0.947 0.931 0.872 0.497 0.822 0.332 0.808
DI2-FGSM 0.999 0.833 0.835 0.832 0.799 0.751 0.324 0.69 0.333 0.711
SI-FGSM 1 0.768 0.764 0.78 0.748 0.686 0.287 0.609 0.33 0.664
Admix 1 0.858 0.855 0.864 0.835 0.774 0.354 0.703 0.33 0.73
TI-FGSM 1 0.716 0.708 0.729 0.691 0.612 0.245 0.572 0.328 0.622
MI-DI 0.999 0.876 0.878 0.866 0.841 0.811 0.372 0.736 0.343 0.747
MI-DI-TI 0.999 0.875 0.876 0.865 0.835 0.812 0.369 0.733 0.342 0.745
MI-DI-TI-SI 1 0.909 0.922 0.905 0.881 0.869 0.44 0.776 0.346 0.783
FIA 0.999 0.894 0.885 0.911 0.892 0.82 0.422 0.83 0.335 0.776
NAA 0.999 0.914 0.906 0.924 0.918 0.854 0.538 0.819 0.336 0.801
ILA 0.995 0.921 0.919 0.926 0.908 0.853 0.459 0.818 0.334 0.793
RD 0.99 0.83 0.816 0.835 0.815 0.733 0.331 0.683 0.327 0.707
VMI 0.999 0.865 0.86 0.866 0.834 0.785 0.356 0.718 0.334 0.735
VNI 0.999 0.867 0.861 0.865 0.833 0.783 0.355 0.718 0.332 0.735
SI-RAP 0.988 0.919 0.918 0.917 0.911 0.882 0.486 0.79 0.336 0.794
GhostNet 1 0.855 0.841 0.854 0.826 0.746 0.316 0.68 0.327 0.716
LGV 1 0.975 0.972 0.976 0.963 0.936 0.551 0.88 0.335 0.843
SWA 0.999 0.955 0.956 0.955 0.945 0.907 0.518 0.827 0.332 0.822
Bayesian_attack 1 0.979 0.981 0.979 0.974 0.946 0.611 0.897 0.336 0.856
LGV-GhostNet 1 0.97 0.969 0.973 0.963 0.931 0.56 0.879 0.335 0.842
surrogate model=Inception-V3
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.432 0.544 0.687 0.568 0.534 0.999 0.17 0.444 0.325 0.522
PGD 0.528 0.661 0.8 0.688 0.642 0.999 0.215 0.536 0.326 0.599
MI-FGSM 0.51 0.613 0.745 0.643 0.604 0.999 0.202 0.55 0.329 0.577
NI-FGSM 0.514 0.614 0.743 0.645 0.601 0.999 0.2 0.54 0.328 0.576
PI-FGSM 0.718 0.809 0.905 0.822 0.792 1 0.325 0.681 0.332 0.709
VT 0.645 0.766 0.873 0.77 0.738 0.995 0.258 0.607 0.326 0.664
RAP 0.69 0.728 0.757 0.733 0.723 0.817 0.402 0.666 0.33 0.65
GhostNet 0.766 0.839 0.93 0.852 0.841 1 0.371 0.692 0.33 0.736
DI2-FGSM 0.643 0.733 0.862 0.735 0.696 0.999 0.255 0.601 0.332 0.65
SI-FGSM 0.482 0.596 0.739 0.617 0.579 0.999 0.195 0.462 0.325 0.555
Admix 0.631 0.747 0.863 0.764 0.73 1 0.274 0.584 0.326 0.657
TI-FGSM 0.429 0.542 0.687 0.569 0.532 0.999 0.168 0.447 0.325 0.522
MI-DI 0.767 0.814 0.917 0.813 0.783 0.998 0.333 0.698 0.342 0.718
MI-DI-TI 0.761 0.812 0.918 0.808 0.779 0.998 0.328 0.69 0.341 0.715
MI-DI-TI-SI 0.823 0.863 0.944 0.849 0.825 0.999 0.393 0.732 0.344 0.752
ILA_BSL 0.435 0.55 0.695 0.557 0.544 0.972 0.156 0.4 0.318 0.514
ILA 0.801 0.881 0.918 0.868 0.872 0.98 0.501 0.765 0.334 0.769
FIA 0.878 0.928 0.951 0.931 0.931 0.979 0.58 0.88 0.342 0.822
NAA 0.832 0.885 0.924 0.884 0.891 0.991 0.579 0.787 0.342 0.791
VMI 0.719 0.815 0.91 0.816 0.787 0.999 0.328 0.673 0.332 0.709
VNI 0.721 0.815 0.91 0.817 0.789 0.999 0.328 0.674 0.333 0.71
SI-RAP 0.911 0.939 0.962 0.938 0.938 0.987 0.621 0.834 0.339 0.83
RD 0.741 0.874 0.962 0.91 0.856 1 0.358 0.719 0.328 0.75
LGV 0.965 0.993 0.997 0.995 0.991 1 0.716 0.937 0.336 0.881
SWA 0.954 0.993 0.997 0.994 0.99 1 0.698 0.929 0.336 0.877
Bayesian_attack 0.97 0.995 0.999 0.997 0.994 1 0.745 0.949 0.337 0.887
LGV-GhostNet 0.893 0.937 0.971 0.952 0.942 0.988 0.516 0.83 0.338 0.818
surrogate model=ResNet-50
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.648 0.801 1 0.88 0.769 0.775 0.306 0.734 0.329 0.693
PGD 0.76 0.893 1 0.946 0.865 0.865 0.389 0.829 0.33 0.764
MI-FGSM 0.727 0.855 1 0.915 0.822 0.832 0.356 0.785 0.338 0.737
NI-FGSM 0.727 0.856 1 0.916 0.825 0.829 0.354 0.788 0.335 0.737
PI-FGSM 0.855 0.933 1 0.965 0.915 0.912 0.48 0.877 0.334 0.808
VT 0.83 0.923 1 0.953 0.902 0.905 0.449 0.863 0.332 0.795
RAP 0.931 0.97 1 0.984 0.962 0.964 0.631 0.936 0.34 0.857
LinBP 0.902 0.976 1 0.993 0.977 0.967 0.659 0.937 0.331 0.86
SGM 0.878 0.955 1 0.986 0.953 0.935 0.605 0.932 0.339 0.843
GhostNet 0.877 0.959 1 0.983 0.952 0.951 0.518 0.919 0.336 0.833
DI2-FGSM 0.832 0.92 1 0.951 0.89 0.904 0.446 0.872 0.338 0.795
SI-FGSM 0.712 0.848 1 0.912 0.818 0.84 0.376 0.753 0.329 0.732
Admix 0.84 0.939 1 0.971 0.913 0.923 0.502 0.866 0.33 0.809
TI-FGSM 0.643 0.801 1 0.879 0.77 0.775 0.31 0.733 0.329 0.693
MI-DI 0.902 0.952 1 0.973 0.926 0.942 0.509 0.906 0.346 0.829
MI-DI-TI 0.9 0.953 1 0.971 0.928 0.941 0.504 0.907 0.347 0.828
MI-DI-TI-SI 0.936 0.968 1 0.982 0.949 0.967 0.609 0.923 0.348 0.854
ILA_BSL 0.625 0.792 1 0.868 0.762 0.778 0.294 0.71 0.322 0.683
ILA 0.877 0.948 1 0.966 0.934 0.935 0.607 0.894 0.334 0.833
FIA 0.863 0.919 0.985 0.933 0.916 0.915 0.542 0.912 0.345 0.814
NAA 0.732 0.81 0.954 0.824 0.808 0.804 0.541 0.732 0.339 0.727
VMI 0.882 0.953 1 0.976 0.931 0.938 0.517 0.897 0.338 0.826
VNI 0.88 0.951 1 0.977 0.932 0.937 0.518 0.896 0.337 0.825
SI-RAP 0.947 0.976 1 0.981 0.973 0.977 0.685 0.904 0.338 0.864
RD 0.818 0.938 1 0.973 0.917 0.923 0.466 0.874 0.333 0.805
LGV 0.986 0.998 1 0.999 0.998 0.997 0.84 0.986 0.336 0.904
SWA 0.972 0.998 1 0.999 0.996 0.995 0.815 0.983 0.34 0.9
Bayesian_attack 0.984 0.999 1 1 0.998 0.997 0.855 0.988 0.34 0.907
LGV-GhostNet 0.967 0.989 1 0.994 0.985 0.984 0.728 0.968 0.343 0.884
surrogate model=DenseNet-121
Target Model  → VGG19-bn WRN-28-10 ResNet-50 ResNeXt-29 DenseNet-121 Inception-V3 PyramidNet272 GDAS WRN-28-10(AT)
Blackbox  Attack↓ ASR ASR ASR ASR ASR ASR ASR ASR ASR mean
I-FGSM 0.49 0.794 0.682 0.942 1 0.514 0.434 0.833 0.327 0.668
PGD 0.614 0.903 0.785 0.982 1 0.618 0.543 0.916 0.329 0.743
MI-FGSM 0.582 0.853 0.752 0.969 1 0.603 0.458 0.877 0.333 0.714
NI-FGSM 0.585 0.854 0.751 0.966 1 0.6 0.458 0.876 0.333 0.714
PI-FGSM 0.714 0.93 0.84 0.989 1 0.693 0.6 0.929 0.333 0.781
VT 0.699 0.916 0.841 0.953 0.999 0.706 0.6 0.911 0.327 0.772
RAP 0.776 0.962 0.879 0.988 1 0.766 0.703 0.951 0.331 0.817
SGM 0.722 0.945 0.885 0.993 1 0.758 0.714 0.961 0.338 0.813
FIA 0.663 0.869 0.772 0.911 0.994 0.678 0.551 0.904 0.334 0.742
NAA 0.709 0.877 0.798 0.92 0.986 0.75 0.681 0.855 0.334 0.768
ILA_BSL 0.467 0.801 0.668 0.922 1 0.5 0.461 0.808 0.319 0.661
ILA 0.773 0.94 0.884 0.977 0.999 0.774 0.702 0.935 0.327 0.812
GhostNet 0.654 0.939 0.843 0.992 1 0.694 0.713 0.949 0.327 0.79
DI2-FGSM 0.693 0.922 0.844 0.986 1 0.692 0.621 0.938 0.33 0.78
SI-FGSM 0.555 0.846 0.734 0.96 1 0.59 0.525 0.844 0.327 0.709
Admix 0.646 0.912 0.81 0.987 1 0.653 0.617 0.908 0.329 0.763
TI-FGSM 0.486 0.793 0.678 0.943 1 0.515 0.433 0.837 0.326 0.668
MI-DI 0.819 0.961 0.91 0.993 1 0.789 0.663 0.96 0.342 0.826
MI-DI-TI 0.815 0.96 0.908 0.993 1 0.786 0.655 0.961 0.342 0.824
MI-DI-TI-SI 0.856 0.973 0.935 0.995 1 0.841 0.743 0.964 0.342 0.85
VMI 0.788 0.965 0.905 0.994 1 0.77 0.658 0.957 0.333 0.819
VNI 0.792 0.964 0.904 0.994 1 0.774 0.66 0.957 0.333 0.82
SI-RAP 0.857 0.972 0.919 0.99 1 0.871 0.797 0.939 0.331 0.853
RD 0.553 0.863 0.757 0.978 1 0.581 0.522 0.901 0.327 0.72
LGV 0.979 0.999 0.997 1 1 0.983 0.945 0.997 0.341 0.916
SWA 0.953 0.998 0.993 1 1 0.97 0.963 0.997 0.34 0.913
Bayesian_attack 0.977 1 0.997 1 1 0.986 0.981 0.999 0.344 0.92
LGV-GhostNet 0.968 0.997 0.991 1 1 0.97 0.901 0.993 0.342 0.907


Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Parsimonious(ECO) 380.4 183 1 207.5 146 1 376.8 183 1 473.2 239 0.994 1119.6 320 0.543 1180.4 359 0.516 875.7 346 0.312
Square Attack 78.6 26 0.986 64.7 26 1 72.4 23 0.987 143.2 43 0.981 1085.2 276 0.534 1135.4 293 0.501 734.6 233 0.812
PPBA 65 16 0.991 43 16 1 63.4 16 0.991 118.5 21 0.995 676.6 348 0.581 743.3 367 0.543 559.4 265 0.837
NES 395.2 190 0.998 267.3 180 1 387.4 192 0.998 600.2 276 　 711.5 280 0.218 821.6 330 0.193 843.5 445 0.942
Bandit 156.3 63 0.988 111.3 54 1 153.2 60 0.99 164.8 83 0.981 1498.2 418 0.384 1451.5 444 0.367 213.6 134 0.523
ZOsign-SGD 421 186 1 257.4 155 1 418.7 178 1 647.2 265 0.993 634.5 195 0.162 614.9 217 0.138 935.2 545 0.84
Advflow 554.7 324 0.993 428.7 261 1 523.9 316 0.993 890.8 448 0.993 897.4 652 0.476 832.5 673 0.453 875.7 761 0.956
SignHunter 152 68 1 106.8 57 1 155 68 1 157 76 1 716.6 229 0.557 683.2 284 0.537 934.2 345 0.774
NP-attack 312.4 24 0.996 239.3 20 1 305.6 24 1 449.3 31 0.994 873.9 542 0.542 843.7 577 0.523 879.3 696 0.954
Subspace   Attack 513.8 298 0.994 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　
CG-attack 178.3 1 1 128.7 1 0.999 169.6 1 1 324.9 12 0.997 1034.7 485 0.584 1187.3 561 0.561 687.3 1 0.947
MCG 38.1 1 1 57.9 1 1 47.6 1 1 69.4 12 1 634.6 96 0.956 698.2 103 0.948 165.7 1 0.962
BASES 298.6 38 1 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　
PRGF 243.7 1 1 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　
Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
SimBA 426.8 185 0.989 378 136 0.994 489.3 185 0.988 657.7 240 0.983 1487.4 987 0.378 1523.2 1013 0.365 1355.6 678 0.456
Square Attack 586.3 185 0.982 464.1 174 0.997 602.6 194 0.982 868.4 236 0.981 1903.6 759 0.341 1935.4 766 0.323 1453.5 345 0.392
PPBA 322.6 122 0.991 279.4 97 0.993 317.4 142 0.992 467.9 157 0.992 1287.4 956 0.443 1357.3 931 0.421 987.4 287 0.475
NES 685.3 345 0.988 374.8 270 1 734.5 367 0.982 1048.7 478 0.981 1093.1 612 0.256 1155.4 630 0.235 1345.6 570 0.91
Bandit 539.8 248 0.986 391 186 1 527.5 238 0.984 863.7 317 0.983 2078.4 1286 0.334 2163.2 1318 0.312 1656.4 645 0.385
ZOsign-SGD 596.3 255 0.995 340.1 275 1 605.3 267 0.994 966.7 365 0.994 400.3 374 0.172 397.5 372 0.144 1853.5 894 0.853
Advflow 886.7 577 0.993 764.5 438 1 834.3 603 0.994 1495.9 762 0.994 674.7 593 0.432 794.3 537 0.417 1743.5 672 0.931
NP-attack 432.5 265 0.994 387 221 1 467.2 275 0.993 668.2 387 0.93 587.2 437 0.548 467.3 402 0.491 1032.8 438 0.942
BABIES 634.6 369 0.991 534.3 278 1 649.2 387 0.992 1036.5 476 0.992 788.6 531 0.579 652.6 478 0.472 1298.7 562 0.595
Subspace   Attack 759.6 532 0.993 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　
BASES 478.9 290 0.994 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　
PRGF 411.7 232 0.995 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　
Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Parsimonious(ECO) 1174.6 678 1 976.5 543 1 1087.7 598 1 1784.2 876 0.932 5556.8 2989 0.086 5876.5 2768 0.053 12567.4 7665 0.122
Square Attack 604.5 389 1 487.9 228 1 584.5 374 1 844.3 579 0.948 9253.4 3985 0.112 9135.4 3867 0.076 12574.4 4233 0.687
PPBA 489.7 267 1 420.6 226 1 465.8 258 1 687.8 488 0.951 7659.2 2364 0.114 7098.3 2084 0.084 15425.8 12448 0.744
NES 2435.6 1365 0.978 2198.4 1198 0.98 2179.6 1298 0.966 2985.4 1698 0.879 　 　 0 　 　 0 10884.5 7567 0.75
Bandit 1098.4 632 1 963.2 618 1 938.7 534 1 2074.8 1249 0.892 　 　 0 　 　 0 7865.4 2345 0.351
ZOsign-SGD 2234.5 1056 0.982 1976.4 877 0.984 2198.5 932 0.975 2764.9 1443 0.917 　 　 0 　 　 0 12357.3 7834 0.76
Advflow 2987.4 1465 1 2634.2 1279 1 2659.4 1375 0.984 3448 1579 0.938 8756.3 3027 0.132 7659.5 2498 0.117 20549.3 11097.8 0.798
SignHunter 856.7 445 1 746.2 378 1 811.7 422 1 1197 642 0.943 8745.7 3298 0.092 7953.7 3064 0.064 10056.7 3245 0.662
NP-attack 2654.1 1278 1 2187.7 1074 1 2439.7 1192 1 3049.3 1489 0.946 8134.8 2642 0.143 7498.3 2187 0.129 19423.3 8754.2 0.782
CG-attack 629.4 479 1 503.6 279 1 598.4 460 1 798.7 578 0.976 4325.5 2149 0.168 3986.2 2041 0.162 7987.4 4114 0.822
MCG+square 157.8 64 1 142.7 48 1 131.1 33 1 283.5 122 0.983 2894.6 944 0.174 2487.1 792 0.173 4768.6 2447 0.845
Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
SimBA 2987.6 865 0.834 3234.5 938 0.873 3387.1 1085 0.821 4392.4 1124.3 0.817 8765.4 3009 0.05 8865.6 2897 0.04 32954.7 21232 0.523
Square Attack 3123.4 992 0.918 2768.9 875 0.923 3410.5 1270 0.913 3839.1 1387 0.884 9866.5 4566 0.07 9657.7 4355 0.05 20675.5 5967 0.36
PPBA 2076 711 0.928 1874.3 1169 0.937 1874.2 842 0.933 2986.5 1265 0.903 8542.7 2679 0.07 8142.8 2458 0.05 26404.1 18768 0.597
NES 2988.7 1236 0.899 2898.4 1365 0.917 3477.1 1542 0.907 4831.9 1984 0.836 　 　 0 　 　 0 31224.6 13645 0.723
Bandit 3098.4 1356 0.914 2987.4 1204 0.921 2876.5 1269 0.91 5149.6 2240 0.857 　 　 0 　 　 0 34234.5 16934 0.421
ZOsign-SGD 3123.6 1546 0.905 3245.6 1456 0.932 3421.3 1732 0.901 5370.9 2379 0.841 　 　 0 　 　 0 32913.7 18455 0.715
Advflow 3876.3 1763 0.938 3587.9 1644 0.944 3689.1 1869 0.941 5935.1 2466 0.883 9982.3 3798 0.08 9148.5 3422 0.06 38769.4 19866 0.758
NP-attack 2398.1 1176 0.941 1984.7 946 0.949 2564.3 1347 0.938 4289.6 1687 0.908 8681.6 3246 0.08 8328.4 3197 0.06 27899.2 12767 0.759
BABIES 2631.8 1533 0.935 2259.3 1138 0.941 2784.9 1639 0.932 4720.3 1479 0.891 8956.3 3572 0.07 8470 3387 0.05 24768.7 14382 0.718



Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
GeoDA 1381 580 0.66 891 431 0.94 1298 543 0.65 1498 602 0.65 811 332 0.49 833 401 0.51 535 345 0.45
Sign Flip 203 120 1 205 120 1 202 120 1 211 120 1 1102 675 0.77 1217 691 0.76 565 345 0.46
Rays 512 339 1 510 338 1 497 318 1 513 339 1 2216 1301 0.79 2171 1245 0.79 876 435 0.75
OPT 2100 1353 0.77 1871 1201 0.77 2067 1241 0.78 2199 1432 0.73 2203 1132 0.25 2241 1356 0.23 0
Sign-OPT 1803 1560 0.96 1866 1578 0.96 1765 1532 0.96 2180 1722 0.96 1878 1603 0.31 1833 1647 0.35 0
HSJA 1906 1025 0.91 2080 1157 0.92 1876 537 0.91 2031 1133 0.9 2751 1566 0.55 2576 1502 0.58 3371 1874 0.49
QEBA method 1332 576 1 1897 1099 1 1277 511 1 1577 687 1 1987 981 0.71 2113 1004 0.72 2455 1976 0.62
Nonlinear-BA 1499 864 1 1954 1127 1 1379 832 1 1699 921 1 2066 1102 0.7 2245 1132 0.71 2684 1421 0.6
PSBA 1255 587 1 1813 1046 1 1198 531 1 1444 621 1 1877 988 0.71 2128 1087 0.72 2114 1551 0.62
CISA 1033 467 1 1643 873 1 864 432 1 1322 562 1 1653 672 0.71 1857 866 0.72 1894 1342 0.63
Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Boundary method 2103 1566 0.05 2275 1803 0.05 1977 1431 0.07 2401 1678 0.04 2002 1277 0.02 2103 1327 0.02 0
Evolutionary method 1877 1115 0.24 2013 1211 0.21 1764 1045 0.26 2033 1341 0.19 1906 1168 0.21 1947 1265 0.2 0
GeoDA 1381 580 0.66 1422 653 0.64 1266 564 0.66 1542 605 0.64 1621 731 0.36 1461 603 0.36 2342 1074 0.21
OPT 1688 1145 0.76 1723 1322 0.75 1455 1021 0.76 1744 1266 0.71 1711 1187 0.2 1787 1108 0.19 0
Sign-OPT 1367 1225 1 1432 1331 0.98 1211 1078 1 1622 1243 0.98 1421 1089 0.32 1429 1131 0.31 0
HSJA 2692 1557 0.9 2911 1688 1 2411 1321 0.9 2877 1768 0.9 3172 2088 0.52 3087 2101 0.53 3190 1873 0.43
QEBA method 1093 482 1 1144 493 1 1023 473 1 1022 431 1 1344 601 0.66 1402 688 0.65 2113 1098 0.54
Nonlinear-BA 1201 767 1 1288 791 1 1109 754 1 1276 753 1 1602 759 0.67 1689 744 0.67 2409 1177 0.53
PSBA 1076 472 1 1099 475 1 997 469 1 1041 760 1 1379 611 0.67 1402 619 0.67 2198 1076 0.54
Triangle Attack 487 225 1 533 247 1 476 221 1 524 288 1 1045 627 0.68 1144 644 0.67 1988 987 0.54
CISA 256 124 1 278 154 1 244 116 1 326 208 1 634 412 0.72 1335 566 0.72 1844 678 0.63
Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
GeoDA 2346.4 1223 1 2413 1206 1 2208 1165 1 2188 1057 1 18784.3 5234 0.287 17636.4 5230 0.247 4568.7 3466 0.785
Sign Flip 2083.4 1298 1 2204 1282 1 1895 1074 1 1786 942 1 16830.3 5534 0.298 16894.4 5345 0.262 9876.7 3475 0.754
Rays 1982.4 1254 1 2063 1194 1 1766 1040 1 1724 967 1 15235.5 5323 0.308 16534.5 5125 0.287 8965.6 3423 0.786
OPT 0 0 0 0 0 0 0
Sign-OPT 12381.3 8654 0.22 12113 8434 0.24 11965 8232 0.22 13422 8407 0.22 0 0 0
HSJA 3890.2 3209 0.88 3743 3084 0.89 3642 3082 0.88 4189 3687 0.89 18745.2 4645 0.2 18637.2 4829 0.18 14567.4 7896 0.666
QEBA method 2134 1586 1 2046 1748 1 2024 1486 1 2455 1764 1 12148 3543 0.36 11976 3756 0.33 9755 3887 0.78
Nonlinear-BA 2286 1656 1 2218 1688 1 2242 1542 1 2676 1874 1 13232 4144 0.34 13134 4254 0.31 9945 4078 0.77
PSBA 2145 1496 1 2076 1562 1 2098 1322 1 2368 1644 1 11986 3458 0.36 11768 3652 0.33 8955 3845 0.78
CISA 1866 1244 1 1678 1342 1 1653 1252 1 2198 1433 1 10887 2982 0.38 10664 2741 0.33 6744 2976 0.79
Model  → ResNet-50 VGG19-bn DenseNet-121 Inception-V3 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Boundary method 0 9821 7774 0.08 0 20362 18678 0.04 0 0 0
Evolutionary method 12381.3 8654 0.22 18741 9611 0.23 126788 9678 0.18 17241 14927 0.2 35354.5 23455 0.1 32356.5 24311 0.11 0
GeoDA 16535.6 7896 0.63 17433 8336 0.67 13987 8431 0.6 13077 6734 0 17586.4 12343 0.41 18585.6 13454 0.43 42345.7 34553 0.32
OPT 18743.4 11823 0.71 17748 12763 0.72 20883 10873 0.69 14790 14092 0.65 36554.3 25466 0.24 37233.3 25565 0.25 0
Sign-OPT 15874.6 12356 0.75 15446 11884 0.75 19123 13114 0.77 13756 13836 0.68 18739.5 12434 0.31 19834.6 13676 0.33 0
HSJA 21345.6 17636 0.74 26695 17338 0.75 21428 14674 0.71 24399 19680 0.7 39534.5 23145 0.59 38737.5 24355 0.6 54345.5 42356 0.43
QEBA method 8774 5458 0.91 11124 5218 0.91 8664 5878 0.9 8667 4798 0.9 14234 6880 0.84 13446 6598 0.84 24565 14224 0.68
Nonlinear-BA 10312 8720 0.89 12347 9983 0.89 10882 8434 0.88 10821 8382 0.87 18771 9880 0.83 17686 9344 0.83 28441 16784 0.64
PSBA 9180 5664 0.91 11349 6293 0.91 9244 5784 0.9 8828 8460 0.9 13998 7891 0.84 13484 7544 0.85 22458 13988 0.68
Triangle Attack 4268 2780 0.92 6904 3678 0.92 4124 2508 0.9 4444 3206 0.89 7743 4986 0.85 7544 4698 0.85 15422 8743 0.67
CISA 3876 2544 0.92 5876 3455 0.92 3866 2187 0.9 3987 2855 0.9 7224 3876 0.86