IOMMU: This costs you 64 MB of RAM

If you have happened to observe similar messages in your dmesg:

[ 0.004000] Checking aperture…
[ 0.004000] No AGP bridge found
[ 0.004000] Node 0: aperture @ 20000000 size 32 MB
[ 0.004000] Aperture pointing to e820 RAM. Ignoring.
[ 0.004000] Your BIOS doesn’t leave a aperture memory hole
[ 0.004000] Please enable the IOMMU option in the BIOS setup
[ 0.004000] This costs you 64 MB of RAM
[ 0.004000] Mapping aperture over 65536 KB of RAM @ 20000000

and you are using AMD-based system w/o AGP video, then my advice is: just leave that as is, do not bother “improving”! Any tinkering with kernel boot options won’t do you any good, as the kernel has already done the best it could.

Just a note: all those messages at the top of the post should only happen if you have 4 or more GiBs of RAM. If you have less than that, and do have those messages – my experience might be inappropriate for your case.

Another note: my BIOS does not have any IOMMU settings (or “Memory hole remapping” settings), so I didn’t try that. You should check if your BIOS has IOMMU-related options first, just as kernel message suggests.

Read on for details.

My system’s kernel at the moment of writing is 2.6.30-1-amd64 (Debian testing), running on MSI’s 790GX-G65 with integrated Radeon HD 3300 video, with 4 GiB DDR3 RAM and a quad-core Phenom II CPU.

Following are the options you shouldn’t try on a similar system:

• passing iommu=noagp kernel boot option. If you run dmesg|grep -i agp, most likely you will see something in the lines of
  

  $ dmesg|grep -i agp
  [ 0.004000] No AGP bridge found
  [ 0.392098] PCI-DMA: Disabling AGP.
  [ 0.392142] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
  [ 0.774847] Linux agpgart interface v0.103

  which tells you that your kernel has automatically discovered the absence of AGP bridge, disabled it and is using 64MB of AGP aperture for IOMMU. On my system, adding that boot option had no effect.

• passing iommu=off kernel boot option. When I did this, there were loads of “nommu overflow” errors during boot, so I didn’t wait until GDM start and rebooted immediately. Clearly, tons of “overflow” errors won’t do any better than a harmless “This costs you 64 MB of RAM”. Also, systems with over 3 GiB RAM do need IOMMU.
• passing iommu=noaperture kernel boot option. On my system, this option has taken 1 more MiB from my available RAM (as reported by free -m or htop), and also enabled software double-buffering IOMMU method (in dmesg, a string similar to SW TIOLB). Performance-wise this is worse than AMD’s GART-based IOMMU, and you loose 1 more MiB of RAM , so don’t do this.
• passing iommu=noagp,noaperture for me had the same effect as passing iommu=noaperture alone, for the reason from point #1 of this list.
• I haven’t tried any of the memaper=[1|2|3|...] options, as I had no problems with the default 64 MiB.
• passing mem=4G may remove that message (I do not recollect if it did for me), but will eat a good deal more of your available RAM (might be because devices are now mapped within those 4G, and not above). So don’t do this.

I’m not aware of any other solutions to the missing 64 MiB RAM message. There were notices of using pci=nommconf boot option, but that is too much for such a simple message – unless you have problems booting.

Below I’m pasting IOMMU-relevant part of the linux-source-2.6.30/Documentation/x86/x86_64/boot-options.txt file, which I believe is the source of ultimate knowledge You should read that file whenever you feel like tinkering with boot options of your AMD64 machine.

I ***highlighted*** some fragments, to support the points from the list above.

IOMMU (input/output memory management unit)
 Currently four x86-64 PCI-DMA mapping implementations exist:
   1. : use ***no hardware/software IOMMU at all***
      (e.g. ***because you have < 3 GB memory***).
      Kernel boot message: "PCI-DMA: Disabling IOMMU"
   2. : ***AMD GART based hardware IOMMU***.
      Kernel boot message: "PCI-DMA: using GART IOMMU"
   3.  : Software IOMMU implementation. Used
      e.g. ***if there is no hardware IOMMU*** in the system and it is need because
      you have ***>3GB memory*** or told the kernel to us it (iommu=soft))
      Kernel boot message: "PCI-DMA: Using software bounce buffering
      for IO (SWIOTLB)"
   4.  : IBM Calgary hardware IOMMU. Used in IBM
      ***pSeries and xSeries servers***. This hardware IOMMU supports DMA address
      mapping with memory protection, etc.
      Kernel boot message: "PCI-DMA: Using Calgary IOMMU"
 iommu=[][,noagp][,off][,force][,noforce][,leak[=]
	[,memaper[=]][,merge][,forcesac][,fullflush][,nomerge]
	[,noaperture][,calgary]
  General iommu options:
    off                Don't initialize and use any kind of IOMMU.
    noforce            Don't force hardware IOMMU usage when it is not needed.
                       (default).
    force              Force the use of the hardware IOMMU even when it is
                       not actually needed (e.g. because < 3 GB memory).
    soft               Use software bounce buffering (SWIOTLB) (default for
                       Intel machines). This can be used to prevent the usage
                       of an available hardware IOMMU.
  iommu options only relevant to the AMD GART hardware IOMMU:
                 Set the size of the remapping area in bytes.
    allowed            Overwrite iommu off workarounds for specific chipsets.
    fullflush          Flush IOMMU on each allocation (default).
    nofullflush        Don't use IOMMU fullflush.
    leak               Turn on simple iommu leak tracing (only when
                       CONFIG_IOMMU_LEAK is on). Default number of leak pages
                       is 20.
    memaper[=]  Allocate an own aperture over RAM with size 32MB<4GB.
                       DAC is used with 32-bit PCI to push a 64-bit address in
                       two cycles. When off all DMA over >4GB is forced through
                       an IOMMU or software bounce buffering.
    nodac              Forbid DAC mode, i.e. DMA >4GB.
    panic              Always panic when IOMMU overflows.
    calgary            Use the Calgary IOMMU if it is available
  iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
  implementation:
    swiotlb=[,force]
                Prereserve that many 128K pages for the software IO
                       bounce buffering.
    force              Force all IO through the software TLB.
  Settings for the IBM Calgary hardware IOMMU currently found in IBM
  pSeries and xSeries machines:
    calgary=[64k,128k,256k,512k,1M,2M,4M,8M]
    calgary=[translate_empty_slots]
    calgary=[disable=]
    panic              Always panic when IOMMU overflows
    64k,...,8M - Set the size of each PCI slot's translation table
    when using the Calgary IOMMU. This is the size of the translation
    table itself in main memory. The smallest table, 64k, covers an IO
    space of 32MB; the largest, 8MB table, can cover an IO space of
    4GB. Normally the kernel will make the right choice by itself.
    translate_empty_slots - Enable translation even on slots that have
    no devices attached to them, in case a device will be hotplugged
    in the future.
    disable= - Disable translation on a given PHB. For
    example, the built-in graphics adapter resides on the first bridge
    (PCI bus number 0); if translation (isolation) is enabled on this
    bridge, X servers that access the hardware directly from user
    space might stop working. Use this option if you have devices that
    are accessed from userspace directly on some PCI host bridge.