Do we need to lay ground rules for performance tests?

[Renoktation]

Well, I am no expert when it comes to microprocessors. But I do understand electronics. And one thing I am not able to get my head around is how does a 35-Watt chip (M1) beat a 100-Watt chip (Intel) in performance tests? Where does that extra 65-Watt go?

I have seen many reviews, but none of them even tries to address this issue. I did some research to find out the differences between the new Apple M1 chip and Intel core 10th generation chips. The first major point of difference was that the fabrication technology and the number of transistors. As a summary, here are the details:

1. Apple M1 - 5 nm process - 16 Billion transistors
2. Intel i7 - 10 nm process - 3 Billion transistors
3. SnapDragon 855 - 7 nm process - 6 Billion transistors
4. SnapDragon 865 - 7 nm process - 10 Billion Transistors

The first question that arises is how can anybody compare a chip with 16 Billion transistors from a chip with 3 Billion transistors, or a quad-core chip from octa-core chips? I understand that there are other factors also that affects performance, but this comparison in itself makes no sense. It's like comparing a 2-stroke engine with a 4-stroke engine.

Even when it comes to applications, tests like CineBench on Mac OS and on Windows 10 are two different applications with identical user interface. They run different number of processes in the background on either OSs to get a task done. Some processes may be security related and so one. How can we use that to compare two devices running on two different OS?

Correct me if I am wrong, but don't we have to set some rules for benchmarking?

My second observation is about "Cost". Apple has built the entire narrative around two key words - POWER & PERFORMANCE. While it has smartly hidden the third key word - COST. I have no idea about the actual cost, but for a company like Apple that now has end-to-end control over its products, it is easier to adjust the increased cost of one component say "M1 chips" into other hardware or software parts, a luxury that is not there in PC world because of multiple vendors.

Yes, current M1 chips outperform some of Intel chips, but there are many other high-end Intel chips that also outperform M1 chips. I think that if we really want to compare ARM with x86 chips, we have to wait till Intel develops 5 nm fabrication process.

 

[xandros9]

You can look at this a number of ways. ARM has always(?) been more power-efficient than x86 in many use cases so it stands to reason that you can eek out more power-per-watt - like the comparison that Apple made when they switched from PPC to x86.

People can definitely compare different chips because raw transistor/core count is just a trait of the chip. 2 stroke and 4 stroke engines differ fundamentally in their operation (despite similarities) and I feel a more apt parallel to draw is the number of cylinders in a 4 stroke engine. But wait, there's more differences too than just comparing cylinder count, so maybe one uses the Otto cycle and one uses Atkinson. Similar, but different. but is it really different enough for a purchaser to care about? It's all going into a PC of some sort for sale.

When benchmarking, it's a tough thing to make absolutely fair but we have to make do with the tools we have ranging from measuring single and multicore performance, video rendering, 3D rendering stress tests... it can certainly be hard to compare, even between OSes, but it's a good frame of reference and still means work is being done faster/slower.

And you imply Apple secretly hid the massive costs of M1, and while I'm sure that R&D was pricey, always is with Apple, it's entirely possible that the M1 is cheaper per machine than the costs of procuring/licensing/using Intel chips. They cut prices after all when announcing the M1 Macbook Air and mini. The problem with switching architecture was always compatibility and R&D, not necessarily cost to manufacture.

And you can say it's not "fair" to compare M1 to chips using 10nm or whatever process, but it doesn't change the fact that the M1 is here today, and is certainly punching above it's weight class and surprising people because of it. Bottom line, they caught Intel resting on it's laurels and deserve the hype. Sure, I'm sure a Xeon (or whatever high-end Intel chip beats M1) could have more oomph, but can it be put into an ultrabook form-factor and use as much power?

 

[Renoktation]

You can look at this a number of ways. ARM has always(?) been more power-efficient than x86 in many use cases so it stands to reason that you can eek out more power-per-watt - like the comparison that Apple made when they switched from PPC to x86.

People can definitely compare different chips because raw transistor/core count is just a trait of the chip.

Comparison of (RISC vs CISC), of (ARM vs x86) and of (M1 vs Qualcomm vs Intel chips) are very different. While RISC vs CISC and ARM vs x86 is more of a theoretical exercise, comparison between chips can be done by benchmark tests and that too if certain conditions are satisfied.

For example, the recently launched Qualcomm Snapdragon 888 chip is more of an advanced mobile chip and should be compared with Apple A series bionic chips because both operate under similar constraints. But it will be incorrect to compare 888 with M1 chips.

When we talk about Intel chips, we have to understand that they follow a top-down approach. They started with high-performance high-power desktop chips and scaled them down to be used on mobile PCs aka laptops. That is their ecosystem - medium laptops to high end workstations. And they guarantee that an app designed for any high-end desktop would run on laptops and vice versa.

Apple and Qualcomm on the other end follow a bottom-up approach. They started with a smartphone chip and scaled it up to fit laptops. Their ecosystem includes low to medium level laptops and desktops. And they guarantee that their apps will run on these ecosystems.

​

So, you see, there are other non-technical constraints & limitations when it comes to processor design. Hence, I feel that we can use these benchmark tests to compare various Intel chips. We can even use them to compare an Intel with an Apple chip, but that will not give a complete picture!

Having said that, Intel's new Lakefield processor is following a bottom-up approach and can be found in Samsung Book and Lenovo Fold.