miserware only delivers a 2.7% power reduction
15/2/10 NOTE: this post has content that is seen as inaccurate by Miserware and you are advised to read all the text before forming conclusions to give Miserware a fair hearing.
I used a power meter to measure the electric draw of the PC using and not using miserware
Interested by some of Alan’s posts at the Open Sourcerer, I’ve been conducting an experiment in recent weeks with the Miserware power saving software that claims to save energy by better controlling CPU management. I run a new (late 2009) pc with an Intel Core 2 Quad Q8200 (4x 2.33) CPU with a 450w PowerCool 80+, and have given it a go. However, I didnt rely on the software’s own claimed power savings, I put an electric meter between the PC’s plug and the socket, and directly measured the power usage over 3 weeks, both with and without the software installed, set at my current domestic electricity rate pkwh.
Here is my data:
| miserware installed? | date | £ of electricity used | £ per day from last reading | £ per day | annual cost | annual saving | |
| no | 19/01/10 14:00 | 0 | pc booted up | ||||
| no | 20/01/10 18:00 | 0.16 | 0.1371 | ||||
| no | 23/01/10 22:00 | 0.62 | 0.1453 | 0.1431 | 52.22 | miserware installed | |
| yes | 24/01/10 08:00 | 0.67 | 0.1200 | ||||
| yes | 26/01/10 19:00 | 1.02 | 0.1424 | ||||
| yes | 30/01/10 08:43 | 1.53 | 0.1428 | ||||
| yes | 08/02/10 08:33 | 2.81 | 0.1423 | 0.1418 | 51.77 | Uninstalled 8 feb | |
| no | 13/02/10 05:12 | 3.53 | 0.1481 | 0.1481 | 54.07 | 2.3 | test stopped |
What I did was to run the PC without the software installed for 4 days from 19-23 Jan, installed miserware from 23 Jan to 8 Feb, then ran the PC again from 8 Feb until today. Congratulations to the miserware support though, when I had issues installing they were right on the ball, and FOC.
The results are pretty unimpressive. Despite a claimed 15% power saving claimed by the software (sudo grep Estimated\ energy /var/log/* ) it actually made very little difference at all. In fact, if you average out the estimated annual usage for the 2 periods without running miserware ((52.22+54.07)/2) = 53.15 and compare that to the annual cost running miserware of £51.77, you see a 2.7% power saving thats worth about £2.30 a year. OK, its a saving, but its not worth a lot to me.
The scary observation is that if you leave a PC on 24/7, its going to cost you over fifty quid a year at current online dual fuel electricity tariff rates!
This morning I received the following email from the CEO of Miserware:
—– Original Message —–
From: “Kirk Cameron”
To: “adrian steele”
Sent: Sunday, 14 February, 2010 05:10:24 GMT +00:00 GMT Britain, Ireland, Portugal
Subject: From MiserWare RE: Your Recent Blog Posting
Dear Dr. Steele,
First of all, thank you for your interest in MiserWare. We recently
read your blog entry entitled “miserware only delivers a 2.7% power
reduction”. We are very interested in independent third party
benchmarks of our software. However, power benchmarks are notoriously
hard to make rigorous which is why our license agreement (see below)
requires written approval of the methodology prior to publishing
benchmarking results. We must respectfully ask you to temporarily
remove your current benchmark posting and all associated links until
receiving written approval for your independent testing methodology.
We would like to work with you to facilitate a publishable,
independent, third-party validation of our software. The goal of
requiring written approval is not to influence the outcome of your
tests, but to ensure that your methodology meets current industry
standards and is accurate, documented, and repeatable. To this end, we
could loan you more accurate power measurement equipment and help with
their experimental use in data collection and benchmarking.
We appreciate your passion for your blog and understand your desire to
give your readers the most accurate information. We hope you will
accept our invitation to work together to conduct an independent
validation of the current beta software.
Sincerely,
Kirk
Kirk W. Cameron, Ph.D.
CEO
MiserWare, Inc.
—————————————
Detailed Licensing Information
Below is the full license agreement you signed on January 19, 2010.
Please forgive the unicode copy/paste errors. The pertinent section to
this discussion is Section 5b excerpted here:
“(b) You are hereby authorized to disclose information regarding the
performance of the Software, provided that such information is (i)
provided by Licensor to You in a software report; or (ii) obtained
using techniques approved in writing by the Licensor.”
—-Begin Full License—-
[full license SNIPPED by me for brevity but available on request]
END
First of all I would like to thank Kirk for taking some time to read the post. I genuinely was not aware of the content of the license agreement when I did the work and wrote the blog post, as I dont have a copy on email from when I signed up to try the software.
I am in two minds as to why you want me to take down the post. It could be because there is a fundamental flaw in my methodology and it significantly misrepresents the software’s performance, and you make similar requests to everybody who posts feedback without getting approval in advance for the methodology, regardless of the outcome of the test. Alternatively it could be that your firm wishes to use a license clause to limit feedback on your product where you feel that such feedback could be perceived as negative to Miserware. I somehow doubt I would have received such a request if my results had shown a 40% power saving though!
As scientists with PhDs we will both be aware of the importance of data overiding commercial interests, and being open and transparent about the methodology of experiments to make sure that they are not flawed and are reproduceable. I dont think we are going to disagree on this!
I am quite happy to take down the content of the post Kirk if we can agree a replacement methodology. I would be pleased to rerun the experiment using a more accurate power meter and I will accept your offer to lend me one to use. I really like the idea of the your software and would like to contribute to its development if I can in a domestic environment. When this replacement methodology is agreed I will remove the content of this post or otherwise indicate that I have temporarily withdrawn it. I am struggling to see where the methodology is flawed though, with the exception of the fact that there are no doubt more accurate power meter available than what I used, but the comparitive data is pretty compelling. Please feel free to advise me on where the methodology is flawed with your expertise in this area.
If the intention is to use your license clause to make me take down the post “temporarily” and not actually undertake any further work, then I have to balance my indicated agreement to the clause (which I was not explicitly aware of as like most people I dont read all the T&Cs of every webpage agreement I make – I dont have the time ) and moral obligation to comply with contracts, against the importance of allowing others who are interested in the subject to be aware of my data and conclusions, even if it is not positive towards your product.
Kirk, please reply through this blog and I will publish your responses in full and unedited.
Best Regards
Adrian
Alan at the Open Sourcerer has made an observation that I didnt state before, which is the CPU is only one of the power consuming units ina PC, and depending on the hardware setup, if the CPU useage is small, even a big reduction in CPU power useage from using Miserware will have comparitively little effect on the total electricity draw (which I measured in this experiment), which what matters.
You can read his post on this at http://www.theopensourcerer.com/2010/01/18/miserware-energy-saving-software-now-cross-platform/#comments
This is the reply from Kirk Cameron of Miserware today. My response is below.
—– Original Message —–
From: “Kirk Cameron”
To: “adrian steele”
Sent: Sunday, 14 February, 2010 21:18:14 GMT +00:00 GMT Britain, Ireland, Portugal
Subject: From MiserWare RE: Your Recent Blog Posting (2nd request)
Dear Adrian,
Thank you for your prompt response. MiserWare software saves
significant energy (up to 35%) while controlling the performance
impact of power management. This is in direct contrast to other power
management techniques (e.g. the PowerNow daemon) that can save energy,
but do not allow user control over the performance impact. The actual
savings achieved will depend upon your system characteristics and
usage. Our software technologies are based upon a decade of
peer-reviewed, award-winning research on improving server energy
efficiency (see http://scape.cs.vt.edu/ ). Our company website
(http://www.miserware.com) contains testimonials from several
independent users that readily received approval for their evaluation
methodology prior to publicly disclosing benchmark results. Our
software has been independently validated privately by Fortune 500
companies in the financial services, government, and media sectors. We
have made the consumer version of our beta software free for personal
use with the hope that hundreds and eventually thousands of people
will use the software to reduce energy waste without noticeably
affecting their productivity. The consumer version you installed
reports CPU-only energy savings using techniques described in the FAQ
accessible via your secure account and pasted at the end of this email
for your perusal.
We are happy to discuss the limitations with your technical evaluation
of our software. Since our software was designed to save energy and
minimize performance impact, any evaluation of MiserWare software
should capture both energy use and performance of the system with and
without our software running. As part of our freeware license
agreement, we require evaluators to obtain written approval of their
methodology prior to (not after) publishing their results since
suspect results in either the positive (great energy savings, no
performance loss) or in the negative (no energy savings plus
performance loss) help neither party to truly evaluate the software. I
hope that you will agree your results were actually positive (3%
energy savings), so we are not trying to bury negative results in this
case. Rather, we want to make sure evaluations of our software adhere
to industry standards. For the average case, after thousands of
benchmark runs on hundreds of systems, we commonly see 20-25% total
system energy savings with limited to no performance impact. Our
software matches the energy used to the load of the system which means
savings will vary with system characteristics and usage. As to your
blog posting and conclusions, your initial assessment is correct.
Specifically,
“there is a fundamental flaw in [your] methodology and it
significantly misrepresents the software’s performance, and
[MiserWare] make[s] similar requests to everybody who posts feedback
without getting approval in advance for the methodology, regardless of
the outcome of the test.”
The industry standard for energy benchmarking is the SPEC benchmark.
Strict adherence to the SPEC methodology would satisfy our
requirements and result in written approval to publish. Among other
things, this benchmark has a defined methodology to capture
performance and energy use and a set of run rules that can be found
here:
http://www.spec.org/specpower/docs/SPECpower-Methodology.pdf
http://www.spec.org/specpower/docs/SPECpower_ssj2008-Run_Reporting_Rules.html
Your techniques violate a number of industry standard run rules.
Firstly, you are not capturing the performance of your system during
both the control runs (without MiserWare) and the other runs (with
MiserWare). Heavy or light use of the system during such intervals can
greatly affect the outcome of your test. For example, running an
intensive graphics application without a high-end graphics card our
software will maximize performance and the system will probably use
maximum CPU energy while the app is running. During another period
while the system is nearly idle our software will minimize energy use
and may save significant energy. For these reasons, the load the
system is under during benchmarking matters immensely and must be
quantifiably measured to ensure fair comparison across samples.
Subjective “interaction” or “user experience” is not typically
acceptable as a performance metric though you probably didn’t notice
any performance degradation when using the software. Keep in mind that
turning your system off is the best way to save energy; MiserWare
software is designed to minimize energy waste while your system is
being used. We do not advocate leaving systems on 24/7 and hope you
will shut down your system when it is not in use for the sake of the
environment.
Secondly, the duration of your tests (i.e. your sample data) does not
appear to be based on standard statistical techniques that consider
variance in the input variables thus giving no indication of the
statistical significance of your conclusions. Variance can come from a
variety of sources including the accuracy of your power meter, its
sampling rate, etc. Were you capturing performance data, variance can
come from measurements in the load variance on the system as well as
the variance in the tools used to capture performance. This may
explain why simply using a more accurate power meter could help, but
is not sufficient to ensure your conclusions are statistically
significant.
I hope that you understand that we greatly appreciate the effort
you’ve undertaken to benchmark our software and our offer to work with
you to meet the SPEC requirements (or another methodology we can
mutually agree to) is genuine. We believe in our software and have a
decade worth of peer-reviewed quantitative data to support our claims
as to its benefits.
However, I am concerned you are in admitted violation of the end user
license agreement you signed. Not reading the license agreement does
not preclude your legal responsibility to adhere to the terms. In
addition to the full agreement you signed digitally on January 19,
2010, the top of the web page with the license agreement provided the
following bulleted list to draw your attention to key portions of the
agreement you were signing:
* You may publish anything output by the program (Section 5.b)
* You are allowed to release benchmark results with prior written
approval of the techniques (Section 5.b)
* You use the software at your own risk (Section 7)’
Since you have publicly acknowledged violating the terms and
conditions of the agreement you digitally signed on January 19, 2010,
I would politely ask you once again to adhere to the terms and
conditions you agreed to and respectfully ask you to immediately
remove your blog posting entitled “miserware only delivers a 2.7%
power reduction” as well as related links that contain benchmarking
information since your techniques have not been approved in writing
and may misrepresent the performance of our software.
With our experience and expertise, we can work with you to reduce the
time, effort and cost required to adhere to the SPEC standard
methodology while still producing statistically significant,
reproducible results. As I mentioned previously, we have no intention
of influencing your conclusions only that the methodology is fair to
all parties. This would allow you to both meet the moral obligation
you have to your readers as well as operate under the terms and
conditions of our end user license agreement.
I appreciate your immediate attention to this matter.
Sincerely,
Kirk
Kirk W. Cameron, Ph.D.
CEO
MiserWare, Inc.
—-Begin Related MiserWare FAQ Entry—-
How are the savings numbers computed?
The savings reported by MicroMiser and ServerMiser ES are processor
power numbers; the consumption and savings numbers are based on the
models of processor power consumption we have created at MiserWare.
Basically, we track how long you spend in each available power state
and calculate savings by comparing against the default highest
performance state for your system. Our techniques and assumptions are
based on a decade of experience and validation using hardware power
meters attached to servers, PCs, and laptops.
The energy, dollar, and carbon savings numbers require some
assumptions about the relationship between processor speed and
processor voltage, and furthermore on some assumptions about your
processor type and the power required to run it. In the current
version of MicroMiser, the assumptions are very simple.
* First, we assume that you are using a 90W processor, typical of
many server and desktop processors available today, although laptops,
some desktops, and now some servers may be equipped with lower power
processors; if you know your processor power consumption, you can
scale the results reported by MicroMiser appropriately to get a more
accurate measure of your savings.
* Second, we assume that your processor has a dynamic power range
of 70%, meaning that 70% of the processor’s power is affected when
processor speed is changed.
* Finally , we assume that the dynamic power changes linearly with
the processor speed, so that running at half the maximum speed would
halve the dynamic power consumption of the processor.
Here is an example. Let’s say your processor has two speeds, 2GHz and
1GHz. Based on our first two assumptions, we assume your processor
consumes 90W at 2GHz with a dynamic power of 90W * 70% = 63W, and a
static power of 90W – 63W = 27W. Based on our third assumption, we
estimate your processor power consumption at 1GHz to be 27W + (63W *
(1GHz / 2GHz)) = 58.5W. So if your processor ran at its highest
performance setting (2GHz) for an hour, it would consume 90W * 1 hour
= 90Wh = .09kWh, and if it ran at 1GHz for an hour, it would consume
58.5W * 1 hour = 58.5Wh = .0585kWh, with a savings of 0.09kWh -
0.0585kWh = 0.0315kWh.
For each time step, the energy savings are calculated in this way, and
the savings are added up over time to yield the savings numbers
reported by MiserWare software. The MicroMiser settings page allows
you to set the assumption for the cost per kWh conversion to dollars
(or in GBP). The conversion to carbon is based on standard techniques
similar to those used by the U.S. Department of Energy. If your system
has multiple physical processors (i.e. multiple CPUs in different
sockets), MicroMiser reports an aggregated total savings achieved for
all physical processors.
Future versions of MiserWare software will include a more
sophisticated power model that takes other factors into account when
calculating power savings and consumption.
—-End Related MiserWare FAQ Entry—-
Thank you for this detailed response Kirk.
I would like to clarify a few things:
1) I did not notice any performance degredation whe I was using Miserware. I should have said this in my initial post, and I apologise for omitting to say this.
2) I agree that the software did reduce my power usage, and therefore using it does derive a benefit. Agreed.
3) I find it perfectly reasonable that others, with different usage profiles and hardware will see much bigger savings than what I experienced.
4) I acknowledge that I may have inadvertently published data in contravention of the license agreement.
The PC I use is on 24/7, as are many, many PCs. This is what I need to earn a living. Indeed Miserware will show a greater impact on PCs that run 24/7 than those that are on occasionally (obviously).
I do not doubt the extensive intellectual input and effort that has gone into the product, but at the end of the day, the only thing that matters is how much electrical power demand is reduced for no (or an acceptable) reduction in performance to the end user. In this respect I stand by my data and approach. If on my hardware, when used by me in a way that suits my work pattern, there is very little benefit in power reduction, then essentially I dont see much of a benefit to me. This is the perspective of an end user who is not a specialist in the field. It is not a ‘like for like’ comparison against your, or the industry standard, methodology.
I accept that my methodology does not meet your industry standard. However, I do not agree that because my testing methodology is not an industry standard then it is not valid. I also note that you have not indicated any specific flaw in my methodology that seeks to establish the total power saving other than it is not the industry standard. My methodology is comparative, and provided that the power meter behaved consistently throughout the tests then I cannt see a flaw in it. With more accurate metering a different result may be obtained. I also offer to give you exact details of my hardware profile so that you can reproduce the results in your own lab,
The methodology I used and my data are clear for all to see and draw their own conclusions. If they think its rubbish, then I’m happy for others to form the view that the method and therefore the results are flawed. My simple approach does not set out to establish what the effect of Miserware is on CPU power usage, that is quite beyond my abilities. However, it does seek to establish what size of saving the software makes when used in combination with all the other variables in my unique working hardware/software/working pattern mix. It may well be that the software saves 15% of CPU usage, but if for example the CPU is only drawing 18% of the total power drawn into the system through the power meter, then that would give a 2.7% indicated total reduction.
There may be other reasons for the result I got, which is why I’m not seeking to discredit your product, its a a great idea, its just that for me, it didnt produce much of a saving. I would like to help it its development if I can but I’m not about to enter into the world of research into CPU power usage.
If you would like to propose a methodology that I can do from home, using my existing power meter or a meter you can lend me then I will happily run further experiments and publish the data. The burden here however does lie with you Kirk to propose a revised methodology that is practicable to do and measures the reduction on my electricity bill from using Miserware. As an end user, this is all I would be interested in.
I propose to fulfil the “moral obligation [I] have to [my] readers” by operating in complete transparency and letting others draw their conclusions from the piece. I acknowledge that I may well be in breech of the user agreement (as I admit I didnt read it), but on balance I feel that the piece is fair and sufficiently clarified and caveated to enable others to judge the worth(or otherwise ) of the work. Your firm is being given unlimited right of reply to say anything you wish about it, and I will continue to do this. Honestly, I’m not out to attack your product just share my experiences!
However, I am not prepared to take the piece down until a revised improved methodology is agreed between us to rerun an experiment that I can do and you are happy with, as I dont think its inaccurate or unfair, and it may be of interest to others. If I do agree to take it down now, then I can see circumstances whereby nothing further will happen from your side and the results will be permanently gone. I am sure that you dont want to be seen to try and enforce a contractual clause to supress real data that for some reason you dont want published (please can you explain why you want it removed?), especially as it is in fact quite positive about your software. The technical support was excellent, there was no noticeable change in performance, and it did deliver a power saving. You should be proud of it. But, for me, using my hardware and my methodology, it only delivered a small saving.
In the long run Kirk, your product will only succeed if it gives a benefit that end users can see in a real world environment, and they will assess this in their user experience (as you correctly say) and in their electricity bill. That is all I measured. If you dont hear it from me, then somewhere down the track others will do just as I have, and indicate the extent of savings in the same manner.
I await advice from you as to how I should rerun an experiment from home, using my power measuring kit or yours, to run a new experiment. If you could simply state for my benefit and others a simple protocol that we can do to test the efficiency of the software ourselves, then this is a great opportunity for you to allow many others to test out your software in the real world. The best improvement I can see would be to run for longer time periods? The equipment available is a willing operator, a PC with an OS that can support miserware, a power meter and a pen and paper to record results.
I look forward to hearing from you Kirk and once again that you for your time in this exchange which I hope is proving useful for us both.
Adrian