Linux kernel 2.6.38 and 2.6.39 power regression workaround

25 Replies

Recently I've been referencing Linux 2.6.38 and 2.6.39 kernel power drain regression in many of my articles. Fortunately Phoronix has found the root cause of this issue and has suggested a workaround for the affected systems. Also because of the Linux kernel 3.0 merge window closed this power regression will not be properly fixed until Linux kernel 3.1 release. In this article I'm bringing you two ways to work around this problem on your laptop or desktop system running Linux kernel 2.6.38 or newer.

Workaround using Sysfs (temporary, use for testing)

Notice:It seems that in the most recent kernel revisions ASPM is disabled on pre PCIe 1.1 devices and must be manually enabled using pcie_aspm=force kernel argument (the second method in this article). So if it happens that when you try Sysfs method you receive Operation not permitted error, then you can skip to the "Workaround by editing GRUB" method.

Sysfs is virtual file system used amongst other things to configure Linux hardware options from userspace. In plain words you can control your hardware options in real time by writing into the simple text files. First lets check the state of things by running following command:

cat /sys/module/pcie_aspm/parameters/policy

The output of this command will probably be something like this:

[default] performance powersave

This means that the default PCIe ASPM (Active State Power Management) profile is selected. This "default" is where the problem lies. To work around Linux kernel 2.6.38 power regression we must force PCIe ASPM to be enabled. For ASPM to be enabled we must make sure that it stays off the "default" and "performance" profiles. This is how you can do it using Sysfs on Ubuntu based Linux distributions:

sudo -i
echo powersave > /sys/module/pcie_aspm/parameters/policy
exit

For other non-Ubuntu based distros you would use su instead of sudo -i. Controlled this way, this setting remains until you change it again using sysfs or until you reboot. This is actually good thing because before enabling this setting at boot time, users should test is PCIe ASPM working fine on their configuration. This is necessary because there are reports of PCIe ASPM causing lockups when enabled on systems with buggy ASMP BIOS implementation. If this option isn't causing problems on your laptop or desktop system you can proceed to the configuring ASPM by editing GRUB or you can use sysfs to enable this option using init script like I've described here.

Workaround by editing GRUB (permanent)

After you've make sure that PCIe ASPM isn't causing problems on your configuration you can apply this workaround for good by editing GRUB configuration like this:

gksudo gedit /etc/default/grub

Now you find the GRUB_CMDLINE_LINUX_DEFAULT line that might look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

You should edit this line to look like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=force"

This way you are passing pcie_aspm=force kernel argument to force PCIe ASPM. This settings will be applied every time you boot your laptop or desktop. At the and you should do following to update GRUB configuration using this command:

sudo update-grub

After reboot you're all set. Your laptop should now spend a lot less juice than without this workaround. Lets all thanks Phoronix for finding the root cause of this Linux kernel 2.6.38 power regression. And feel free to share results for your configuration. Cheers!

DevGenii

A quality focused Magento specialized web development agency. Get in touch!

25 thoughts on “Linux kernel 2.6.38 and 2.6.39 power regression workaround”

Paresh Mathur June 29, 2011 at 16:19:49

Hey,

I was just trying out your work around (I get 24.4W consumption on ubuntu while I get 16ish Watt on windows). Here is the problem

rick@Abigail:~$ cat /sys/module/pcie_aspm/parameters/policy
default performance [powersave]

It marks powersave as the profile. But That does not make any sense. Can you help?

Reply ↓
Marko AuthorJune 29, 2011 at 20:37:16

@Paresh Mathur

Hi! If I’ve understood correctly, your system isn’t affected by the Linux kernel 2.6.38 kernel bug if your PCIe ASMP isn’t on performance or default. The thing is that Linux has troubles reaching Windows power usage because of tweaks hardware manufacturers take to caterer Windows so Ubuntu power usage is always a bit higher.

To be sure are you still affected you could do some testing with solid kernel like 2.6.32 and using 2.6.38 and than you will know for sure. Here you can find my article on how to compile kernel from kernel.org using something like 10 steps it is really easy and you can easily test any kernel you want:

http://www.techytalk.info/compile-upstream-kernel-org-ubuntu-debian/

Cheers!

Reply ↓
Rahmadi June 30, 2011 at 05:47:04

Hi,
There was error when I typed “echo powersave > /sys/module/pcie_aspm/parameters/policy”. The error was “bash: /sys/module/pcie_aspm/parameters/policy: Permission denied”. I tried adding “sudo” at the beginning of the command, but the same exact error showed up.

Can you help? Thanks.

Reply ↓
guyr June 30, 2011 at 07:56:36

@Rahmadi
the sudo command apply only to echo powersave
So i suggest these 3 lines :
sudo -s
echo powersave > /sys/module/pcie_aspm/parameters/policy
exit

Reply ↓
Marko AuthorJune 30, 2011 at 09:59:49

@guyr
@Rahmadi

Sorry guys I’ve made a mistake because I was doing things as root on a Debian PC. Now it should all work at least on “sudo” distros 😉 Thanks

Reply ↓
Rahmadi June 30, 2011 at 11:47:22

Hi guyr,
When I typed line number 2 from your suggestion above, another error showed up: “bash: echo: write error: Operation not permitted”.
I could load the file using sudo gedit, however I could not save it, the error was: “Could not save the file /sys/module/pcie_aspm/parameters/policy. You do not have the permissions necessary to save the file. Please check that you typed the location correctly and try again.”

I am using Ubuntu 11.04 (natty), kernel 2.6.39 from ppa.

Thanks for your answer.

Reply ↓
Marko AuthorJune 30, 2011 at 11:54:35

@Rahmadi

Gedit isn’t necessary because I’ve changed the article using quyr pointers (thanks guyr) so it shouldn’t give you any errors. If you really want to do this using gedit, you should use gksudo gedit /sys/module/pcie_aspm/parameters/policy and then drop powersave inside and save. Hope it works 🙂

Reply ↓
Rahmadi July 1, 2011 at 14:52:47

Hi Marko,
I did tried the newest method above (using sudo -i), however I still encountered error: “-bash: echo: write error: Operation not permitted”.

Thanks.

Reply ↓
Marko AuthorJuly 1, 2011 at 15:06:45

@Rahmadi

That is really strange because after you enter sudo -i on Ubuntu operating system you are practically root and there is nothing that can answer “Operation not permitted” to your command. The only thing could be that /sys/module/pcie_aspm node doesn’t exist on your system. The reason could be that you are running pre 2.6.35 kernel that doesn’t support pcie_aspm or that for some reason pcie_aspm isn’t supported on your machine. There is also possibility that you’ve messed things with sudoers file (man sudoers).

Here’s copy/paste for sysfs procedure from the terminal on my Ubuntu Maverick system with custom 2.6.38.8 kernel:

marko@marko-pc:~$ cat /sys/module/pcie_aspm/parameters/policy [default] performance powersave marko@marko-pc:~$ sudo -i [sudo] password for marko: root@marko-pc:~# echo powersave > /sys/module/pcie_aspm/parameters/policy root@marko-pc:~# exit logout marko@marko-pc:~$ cat /sys/module/pcie_aspm/parameters/policy default performance [powersave] marko@marko-pc:~$

Reply ↓
nukleus July 1, 2011 at 17:01:59

@Marko
Hi Marko,
I’m using a gentoo system using 2.6.39 at the moment, and I do get the “Operation not permitted” error when logged in as root or after becoming root using sudo -i, just like Rahmadi does.
The file does definitely exist and is readable, perms are 0644 (-rw-r–r–). In my kernel config, I have
CONFIG_PCIEASPM=y
#CONFIG_PCIEASPM_DEBUG is not set

Any idea?

Reply ↓
Rahmadi July 1, 2011 at 17:03:38

@Marko,

I am confused myself.
The file /sys/module/pcie_aspm/parameters/policy does exist. I can open it using sudo gedit (the content is: “[default] performance powersave”), but cannot save it.
I am using kernel 2.6.39 from ppa.
I never touch sudoers file.

Really strange.
Anyway, thank you very much for your efforts and answers Marko.

Reply ↓
Tobbs July 1, 2011 at 18:30:45

https://bbs.archlinux.org/viewtopic.php?id=120640

this could be the reason..

Reply ↓
Marko AuthorJuly 1, 2011 at 21:04:25

@nukleus
@Rahmadi
@Tobbs

Thanks Tobbs, this could be it. It sounds that on some hardware you must first force aspm by editing GRUB and then you can control aspm by using sysfs. This is really dumb and I don’t consider this as a feature, this is a bug. If someone else confirms this because on my hardware it works even with the 2.6.39 kernel, i will add it to the article as notice. Again, Tobbs thanks for this.

Reply ↓
vitto July 2, 2011 at 17:23:59

@Marko

echo powersave > /sys/module/pcie_aspm/parameters/policy
-bash: echo: write error: Operation not permitted

also on my acer travelmate

Reply ↓
Marko AuthorJuly 2, 2011 at 18:25:25

@vitto

Hi, thanks for reporting with your results. For more info you can check this link:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/760131/comments/99

It looks like in recent kernel revisions ASPM is disabled on PCIe 1.1 devices and must be manually enabled using pcie_aspm=force kernel argument (the second method in this article).

Edit: I’ve noticed a lot of ASPM related commits in Linux 2.6.39 changelog where kernel devs complain on ASPM on certain hardware:

http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.39

That could explain this kind of behavior.

Reply ↓
Rahmadi July 4, 2011 at 14:03:36

I found a way to set “pcie_aspm=force” temporarily:
-Restart your computer.
-When the computer restarts, immediately hold Shift, and the boot menu will appear.
-Press “e” to edit the first entry.
-Find text “quite splash”, add “pcie_aspm=force” (without quotes) after that text.
-Press F10 to continue boot with the new setting.
-To check, type “cat /sys/module/pcie_aspm/parameters/policy” at terminal.
-In my experience, when you charge the battery (by pluging in the cable), the setting resets to default.

I am sorry I cannot provide you the source of the information above, I forgot the link.

Reply ↓
Marko AuthorJuly 4, 2011 at 16:15:58

@Rahmadi

Hi Rahmadi! The same procedure can be used when GRUB method is used without testing if something goes wrong. Using the procedure you wrote in this situation way we can remove “pcie_aspm=force” from kernel line to be able to boot and then fix GRUB permanently. Thanks for the procedure and great job 🙂

Reply ↓
bug? July 8, 2011 at 21:47:41

Has the kernel team confirmed this to be a bug? When I read about it the kernel team said they were aware of it, and that it wasn’t a bug at all.

Enabling this can cause some machines to crash without warning, if it is off leave it off.

Reply ↓
Marko AuthorJuly 8, 2011 at 21:54:02

@bug?

If I’ve understood correctly, this isn’t a bug. The problem is that pcie aspm can’t be reliable enabled on all configurations due to buggy aspm implementations. But I see no reason for disabling sysfs control?

Reply ↓
Virus August 7, 2011 at 09:09:05

Hi guys,

The problem lies deep in the Kernel and it can’t be fixed even with this fix. It is better to just wait for the Kernel 3.1. I think it is not worth fixing or spending time on it to find a work around. Not like it is going to stay the same for next 5 years like windows. Linux CHANGES fast.

And you might wanna fix this line “At the and you should do following to update GRUB configuration using this command:” “:D

Reply ↓
Marko AuthorAugust 7, 2011 at 11:24:42

@Virus

Yes from my testing it doesn’t work for everyone and when it works it doesn’t completely fix problem. People should wait for 3.1 kernel as you said. My fix was to compile kernel 2.6.37 on all of my Linux boxes. You can find out how to do this in my “Compile upstream Linux kernel from kernel.org on Ubuntu or Debian” article 😉

Reply ↓
Todd October 31, 2011 at 05:35:01

Hi i have Toshiba L300 – PSLB8E (1A6) model, with H2O Bios inside v2.20, i think i have overheating problems and Fn keys dont work :SS and on battery power i have 30min, on Windows 7 1h :S what is whrong why Windows is better than Linux ?

Reply ↓
1. Todd October 31, 2011 at 05:36:23
  
  andhow can i fix it :S i know this problem so so so so so old :S
  
  Reply ↓
  1. Todd October 31, 2011 at 05:39:08
    
    I use now Ubuntu 11.10, i tried Fedora 16 beta, Jolicloud 1.2, Open Suse…
    
    Reply ↓
    1. Florin November 8, 2011 at 11:28:22
      
      Maybe you should follow the instructions in this article. In my
      Toshiba (C650, Intel i3) it quite fixed my battery consumption.
      
      I had first to set the Grub and then the powersave settings:
      STEP1:
      sudo gedit /etc/default/grub
      Insert in the line “RUB_CMDLINE_LINUX_DEFAULT” pcie_aspm=force, it
      should get like this:
      GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash pcie_aspm=force”
      Update your grub:
      sudo update-grub
      Restart to apply the settings. In my Oneric, STEP2 did not work
      without STEP1.
      
      STEP2, test for temporary powersave:
      sudo -i
      echo powersave _ /sys/module/pcie_aspm/parameters/policy
      exit
      It should not echo any error.
      STEP2, permanent powersave (it will reset on restart/suspend):
      sudo gedit /etc/rc.local
      echo powersave _ /sys/module/pcie_aspm/parameters/policy
      Restart.
      
      Check the powersave settings:
      cat /sys/module/pcie_aspm/parameters/policy
      It should print something like this:
      default performance [powersave]
      
      Reply ↓