x86emul: V{,P}{COMPRESS,EXPAND}* can (wrongly) trigger assertion

AFL has pointed out that the op_bytes-is-not-0 assertion in common SIMD
handling can trigger for these insns. Indeed when the (relevant part of)
the controlling mask register is 0, no memory is accessed at all. Leave
op_bytes unaltered in this case, to engage the short-circuiting in common
SIMD handling when fault_suppression is true and op_bytes is 0.

While there also correct a related typo in the test harness.

Fixes: 65f82d4ce1 ("x86emul: support AVX512{F,_VBMI2} compress/expand insns")
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Acked-by: Andrew Cooper <andrew.cooper3@citrix.com>
master commit: cf90c8a48a
master date: 2026-07-20 09:50:00 +0200

tools/tests/x86_emulator/predicates.c

@@ -1945,7 +1945,7 @@ static const struct evex {
     { { 0x83 }, 2, T, R, pfx_66, W1, Ln }, /* vpmultishiftqb */
     { { 0x88 }, 2, T, R, pfx_66, Wn, Ln }, /* vpexpandp{s,d} */
     { { 0x89 }, 2, T, R, pfx_66, Wn, Ln }, /* vpexpand{d,q} */
-    { { 0x8a }, 2, T, W, pfx_66, Wn, Ln }, /* vpcompressp{s,d} */
+    { { 0x8a }, 2, T, W, pfx_66, Wn, Ln }, /* vcompressp{s,d} */
     { { 0x8b }, 2, T, W, pfx_66, Wn, Ln }, /* vpcompress{d,q} */
     { { 0x8d }, 2, F, R, pfx_66, Wn, Ln }, /* vperm{b,w} */
     { { 0x8f }, 2, F, R, pfx_66, W0, Ln }, /* vpshufbitqmb */

xen/arch/x86/x86_emulate/x86_emulate.c

@@ -6174,9 +6174,11 @@ x86_emulate(
             ASSERT(op_bytes == n * elem_bytes);
             op_mask &= ~0ULL >> (64 - n);
             n = hweight64(op_mask);
-            op_bytes = n * elem_bytes;
             if ( n )
+            {
+                op_bytes = n * elem_bytes;
                 op_mask = ~0ULL >> (64 - n);
+            }
         }
         goto simd_zmm;