x86/hvm: reduce the need to flush caches in memory_type_changed()

The current cache flushing done in memory_type_changed() is too wide, and
this doesn't scale on boxes with high number of CPUs.  Attempt to limit
cache flushes as a result of p2m type changes, and only do them if:

 * The CPU doesn't support (or has broken) self-snoop capability, otherwise
   there would be leftover aliases in the cache.  For guest initiated
   memory changes (like changes to MTRRs) the guest should already do a
   cache flush.
 * The IOMMU cannot force all DMA accesses to be snooping ones.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Jan Beulich <jbeulich@suse.com>
Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>

CHANGELOG.md

@@ -12,6 +12,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
    - For ARM32 and ARM64, GCC 5.1 and Binutils 2.25
  - Linux based device model stubdomains are now fully supported.
 
+ - On x86:
+   - Restrict the cache flushing done as a result of guest physical memory map
+     manipulations and memory type changes.
+
 ### Added
  - On x86:
    - Option to attempt to fixup p2m page-faults on PVH dom0.

xen/arch/x86/hvm/mtrr.c

@@ -782,14 +782,22 @@ HVM_REGISTER_SAVE_RESTORE(MTRR, hvm_save_mtrr_msr, NULL, hvm_load_mtrr_msr, 1,
 
 void memory_type_changed(struct domain *d)
 {
-    if ( (is_iommu_enabled(d) || cache_flush_permitted(d)) &&
+    if ( cache_flush_permitted(d) &&
          d->vcpu && d->vcpu[0] && p2m_memory_type_changed(d) &&
          /*
           * Do the p2m type-change, but skip the cache flush if the domain is
           * not yet running.  The check for creation_finished must strictly be
           * done after the call to p2m_memory_type_changed().
           */
-         d->creation_finished )
+         d->creation_finished &&
+         /*
+          * The cache flush should be done if either: CPU doesn't have
+          * self-snoop in which case there could be aliases left in the cache,
+          * or (some) IOMMUs cannot force all DMA device accesses to be
+          * snooped.
+          */
+         (!boot_cpu_has(X86_FEATURE_XEN_SELFSNOOP) ||
+          (is_iommu_enabled(d) && !iommu_snoop)) )
     {
         flush_all(FLUSH_CACHE);
     }