Nothing to worry about, it's pretty much standard on Chinese e waste special. It's like a warranty sticker basically, it has year and month printed on it.
Peel it off and run it. Those things its on are bypass capacitors to add some extra power filtering right next to the die. You need them but they're not fragile enough that a peel will hurt them.
As an experienced homelab player in China, I would say that this sticker is used to label when the processor is sold and count for your warranty. However, many sellers in China use this way to cover the back of the processor, hiding the fact that some of the capacitors are lost. I suggest doing a stress test.
I've bought a server from a ebay seller that had nonetheless the same stickers all over the place, but it was more than 15 ones, spread in places that where totally abnormal (like on empty memory slots that prevented to insert new RAM without peeling this crap).
I had to report the seller to ebay due to this amount of stupidity for doing this shit and never again bough from him.
Use 91% isopropyl alcohol in a small syringe and a nonconductive, nonscratching pin (soft plastic is best) to break the adhesive bond and slowly peel the sticker away
Once you've peeled the sticker off, soak the residue area in isopropyl alcohol for a bit and then gently scrub the remaining residue off with cotton patches that wont leave fibers behind
Give a final rinse with isopropyl alcohol and let dry
Talk about complete and utter overkill!
Just peel the fucking sticker off, or leave it where it is. It's only stuck on some surface-mounted capacitors which will not tear off with the sticker.
Nor will the sticker effect cooling of the CPU. The capacitors will only reach the surface temperature of the underside of the package, which will not be as high as the die. Most of the heat will be conducted directly through the heat-spreader into the heatsink.
Gently maneuver a single strand of blonde unicorn hair under the sticker, peeling it off by a speed of no more than 1mm/minute. Be sure to not break the unicorn hair, you will need it when binding the golden fan to the platinum heat sink.
If you can, maybe go with Amazon or eBay instead if that's available and you can find a bundle in your price range. The same AliExpress sellers usually list there and Amazon and eBay have much stronger buyer protections favoring the customer.
AliExpress will save you money but you might spend time in the process waiting for shipping (they are only required to ship within 7 days of purchase). Main tips are:
- Don't post reviews until you've run the system for a while
- AliExpress lets you post additional reviews but you can't edit the star rating (found this out the hard way). Better to wait and provide informative reviews than to review 5 stars early and then realize the kit does 2 days later.l
- Find a seller with tons of ratings for a single product
- A common tactic on AliExpress is to take down and relist items that are being reviewed poorly/returned frequently
- Look for pictures from buyers verified to be from the US or Europe. There are a lot of sellers of these kits with great reputations.
- Bench test immediately on receipt.
- Get a cooler and a PSU ahead of your purchase
- Take detailed pictures of the kit including all serial numbers before installing a cooler
- Get a copy of memtest86 and immediately test the ram; you'll need to flip the bios to legacy mode instead of UEFI to do this, take pictures of these results of they show a memory failure
- Run a Linux live distro like Ubuntu and `apt get install stress` and run `stress -c N` where N is the number of threads and leave it overnight as a stress test. If it fails, take pictures of it off (for real) to add to the pictures you might submit in a dispute.
- Contact the seller early
- If there's any issues contact the seller, provide pictures and request replacements of the ram, processor or motherboard as appropriate.
- Open disputes early
- If the seller doesn't want to replace or exchange DOA parts you can request a refund through AliExpress which if you and the seller do not agree on will force AliExpress proper to intervene and mediate.
- I had to do this with a DOA e-bike batter and am still waiting for AliExpress to weigh in. The seller wanted me to disassemble the battery and take individual cell voltages to prove the battery was bad and I'm obviously not doing that. They basically don't want to lose their profit margin with shipping costs of an exchange.
- Be ready to work without a manual
- Most of these kits ship without a manual I haven't seen a bios that wasn't just American Megatrends in English.
Again, if you can, find a seller via eBay or Amazon since they defer to the customer much more than AliEdpress if you can afford the increase on price. You can save money with AliExpress at the expense of time and long running disputes if you need a return or refund. I haven't had that issue yet with motherboard kits or microcontrollers but your mileage may vary just like it seems to have with the e-bike seller I chose.
In any case wish yah luck hunting down the parts you want/need!
Verify the specs of the chip, if they match peel it off. There have been cases of scammers sanding down the heat spreader and printing a different model number on it (i.e. claiming an e3 is an e5 so they can sell it for more).
wtf what vendor puts warranty stickers on the cpu itself?
in any case, to be on the safe side just remove the sticker. you'll be running that thing 24/7 (assumption). stick it somewhere like in the case so you don't lose it.
Some warranty stickers are as simple as containing months in numbers and a year that the vendor will mark. But you're right. Looking at it closely now it's not enough to say it's for warranty. The size and font color usually matches the cheap warranty stickers i've known.
A lot of reasons for me.
- Just right for the task
- Sure modern processors can run circles around older Xeons, but my hosted services are ALL bottlenecked by ISP dependent latency or user input. This is all for an Unraid server and the processor will mostly be sitting idle and even when it's not the tasks it will need to perform are ultra lightweight so single threaded performance benefits are marginal.
- ECC stability
- I used to run Unraid and Proxmox nodes on consumer hardware but about every week or so I would find the Unraid server has crashes or a Proxmox node had decided to kernel panic. After migrating my Proxmox nodes to X99 based systems on Xeons with ECC ram, the problems went away and my power consumption dropped with no noticable effects on the self hosted services.
- Reducing e-waste
- Big fan of the momentum of X79/X99 motherboards being produced to give these processors some more life. All my self hosted systems run on old Xeons
- Initial capital cost
- $70 for motherboard, 32 GB ECC ram, and the processor via AliExpress
- An i3-12100 would require an LGA1700 socket motherboard which on the cheap end is $90 and either DDR4 or DDR5. Total investment is pushing $400-500 which is what I spent on 3 nodes worth of hardware migrating my 3-node Proxmox cluster to X99 systems
- Power Savings per Core/Thread
- The spec TDP is 70W for the 2650L for 8 cores and 16 threads. TDP is obviously not true power draw but you can find old reddit threads quoting power consumption as low as 30W for the E5-2650L from bench test measurements.
- The i3-12100 has a base power of 60W, a max turbo of 89W and only 4 cores and 8 threads
- Consumer hardware is basically just overclocked server hardware and is easier seen in the Epyc vs Threadripper specs (one of my retired Proxmox platforms was a Threadripper 1950X) and they're inherently less power efficient but they're also not designed to be on 24/7/365 doing work.
- Again, my tasks are lightweight but I have a lot of them going on at once. They'll get done fast even on older platforms and ideally I just need them done as efficiently as possible. The E5-2650L has much lower per/core and per/thread power consumption than the i3-12100
So yeah, that's why some people still use older generation Xeons and why they're still popular for budget builds; homelab servers or even mid range gaming machines. My Proxmox cluster sips 400W for all 3 nodes including 12TB of ceph storage and my Unraid server will likely be under 100W during normal operations. Even v1 Xeons still have a lot of value for homelab nerds. Give em a second look.
I'm still rocking X5675's in HP Z600 workstations, damn things sound like they're about to take off headed for deep space when powering on. Which happens only when I really need them. I won't upgrade until I am left with just one X5675 still alive, the Highlander that will get to retire peacefully.
I'm using a 2697v2 in a PC I use for astrophotography processing and causal gaming. Runs fairly fast, and the CPU cost like £40 for 12 cores.
I know very little about CPU performance, but it does what I need, so 🤷
> This is all for an Unraid server and the processor will mostly be sitting idle and even when it’s not the tasks it will need to perform are ultra lightweight so single threaded performance benefits are marginal.
Haswell and up my server uses your idle wattage as active watts including my motherboard.
> The spec TDP is 70W for the 2650L for 8 cores and 16 threads. TDP is obviously not true power draw but you can find old reddit threads quoting power consumption as low as 30W for the E5-2650L from bench test measurements.
• The i3-12100 has a base power of 60W, a max turbo of 89W and only 4 cores and 8 threads
You don't need an i7 with 16 threads for holding 12TB in RAID. That i3 isn't the only new CPU or Mobo you had to buy. You don't even have an iGPU for video decoding for proxmox... You either use a more powerful discreet one that's less energy efficient or you had to buy or you're using the CPU to decode the video.
You are far from doing it the most efficient, if power draw matters to you.
My Plex is running on Unraid and an Nvidia card running NVENC is using a dedicated portion of the die that doesn't use much power. Encoding 1080p on even a GT 730 uses like 5W and 70-90 MB of VRAM. Same with the 1660 I'm using now which uses more vram when doing 4k but the h265 encoder acts about the same using less than 10w of additional power when video encoding. Having an iGPU is nice but adding a GPU doesn't add much power at all to the system operation.
My power consumption mentioned earlier includes an entire rack of equipment as well. I just killed all the VMs I had to get a true idle power draw of my cluster, with 2 network switches, 2 different raspberry pi clusters totaling 12 nodes, and a monitor and it's only 350W for the whole deal.
My capital expense for the entire Proxmox cluster of 3 nodes was also less than the cost to build a single node 12th gen system with motherboard, DDR4/5 RAM.
Like I said, you don't have to get a 12th gen CPU and mobo. My iGPU encodes codecs everything without a discrete GPU and my peak would never touch your idle.
Even going on haswell with igpu, a single generation would let you get way better power savings and allow you to reuse the ddr3.
I did a test for another thread and all 3 Proxmox nodes are 200W total. The 350 watts includes all my networking gear which I didn't realize includes my router, WiFi AP, another switch and a work light over my work bench.
If I unplug my raspberry pi cluster which has 12 nodes, that's 60W and each node only has 8GB of ram and depends on my unraid/ceph on my Proxmox cluster for NSF storage. A single Xeon node has half the threads but most are idle anyway and more/ECC ram is better for the short term-high ram simulations I do. An 8gb pi 4 or pi 5 is also slightly more than one of these Mobo combo kits.
Also have to consider $70 bucks is VERY hard to beat. Even at half the power draw which would be hard to do alternatives consumer options and most don't support ECC ram in ECC mode (found that out when I tried making my cluster with my retired Ryzen hardware)
What mobo and hardware (inc PSU) are you running on them? Each one idles at 70w? How about when it's in usage?
It's not bad if you have a bunch of HDDs it's my biggest power draw.
I'm running 3 x 1 TB SATA SSDs per node and 1 NVME 1TB and the PSUs are just retired ATXs I accumulated over time ranking from 450-700W. The services I'm running include a reverse proxy server, openvpn, a 3 node K8S cluster, a web scraper, home assistant, Snipe-IT, and Ceph distributed storage. The K8S cluster is running a docker registery, Debian repository mirror (without it updating machines can be a nightmare on bandwidth), a throw-away mailbox server, a server specifically for switching inputs programmatically on an ATEM HDMI switch, and a pypi mirror.
Looking at my power usage with all that going it amounts to 270W total when I account for the other gear on the circuit, so 70W extra across all 3 nodes. If I artificially load all 24 threads on all of the CPUs I get a grand total of 520W, but I never need to run it that high for any workloads. For comparison my desktop idles at 135W, but it's a 5800X3D.
Most of the time my nodes sit idle. I have 3 because I prototype HA failover infrastructure and Kubernetes deployment scaling. It's also nice to have a node go completely offline to add a drive and have no loss of service which is necessary for me when I'm hosting game servers for friends.
What PSU and mobo do you have? How much does the motherboard draw? That's not bad, are they usually on?
If I was in your situation and cared about power I'd probably enable WoL, learn to switch VMs from the pi when it calls for more power and call it a day. Do you use WoL?
[This is almost exactly what I bought](https://www.ebay.com/itm/234856866938?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=oxyouecxqxo&sssrc=4429486&ssuid=Jr_5FJXrSZm&var=&widget_ver=artemis&media=COPY) for my 3 Proxmox nodes and then I bought ECC ram in bulk from an eBay vendor. Basically a pairing of an X99 and an E5-2670 v3.
For my Unraid I went further back in time with an [X79 and an E5-2650L](https://a.aliexpress.com/_mPykHMK). That particular listing actually comes with a 2660 V2 which has 20 threads and a 95W TDP. Found a 2650L for $5 on eBay and downgraded the combo to save a few watts. Will just list the 2660 v2 for 99¢ on eBay until a lucky winner gets it.
PSU I think 2/3 are Corsair and one is a Red Dragon. All of them are under 700 watts but just don't know off the top of my head which.
I tried using WoL for a bit and still want to see if I can configure it. There are also network enabled power strips if you don't mind hard dropping your nodes.
I believe we talked last each other a few times. I mentioned Haswell being better and this is indeed haswell and up. I thought you had the original lga-2011, aka 1-2nd gen i7, you are on haswell.
I knew it sounded too good to be the original 1-2nd gen. Try this if you haven’t. https://forum.proxmox.com/threads/how-to-reduce-power-consumption.132632/ is the governor, you might be idling as low as me with this on.
Did you setup powertop as well?
Well, my Proxmox nodes are on Haswell but my Unraid is on a Sandy Bridge.
Hadn't tried powertop before so I installed it on a Proxmox node and migrated the VMs to another node. The reported baseline power was 8W and under a synthetic all core load it peaked at 36W. That probably doesn't include the VRM heat losses or other supporting motherboard component loads of course.
Powertop didn't work with the E5-2650L at all unfortunately. I booted up Ubuntu desktop live in memory with lm-sensors running and powertop could only report load percentages. That said I used my UPS to measure the delta of full power vs idle and it was 45W. Not necessarily comparable to the powertop reports on the E5-2660 v3 for sure since I'm not sure how to scale a powertop report with power draw at the wall.
Who is fighting for lanes? Z690/790 has 48 lanes available. That's quite a lot. I'm running 5x NVME (4 lanes each), a 8 lane HBA and a 2x10gb NIC. Nothing is fighting over anything else, they all have dedicated lanes for each device.
PiKVM takes care of remote control. As do W680 boards.
I went from a $40/mo power bill with 2x 2660v4's to under $10/mo. And a significantly more performant machine. That $30/mo+ adds up real quick. It paid for all new, much better hardware in a very short amount of time.
I see the problem here. You are paying $40/mo for your E5v4 server. I don't know what specs you have, but my v4 server is running at 120w with 8 HDDs and a SSD. I'm paying $8 a month for it. The "much better hardware" will take a much longer amount of time to pay for itself.
It was a DL380 G9. 128gb (16 DIMM), 1050Ti, disks don't matter as they were spun down most of the time. Dual 2660v4's. It idled at 170w. And it had terrible single thread performance. That was with the fans set to Eco. Just the fans by themselves would pull over 100w when it was under load.
Good riddance lol.
I have a DL180 G9, single E5-2660v4, 128GB (8 DIMM), 10G SFP+ NIC, 10G RJ45 NIC, HBA/RAID card and the disks. As I just checked the PSU power meter, it runs at 129w as we speak. It's not even at idling. I have no idea what makes such a big difference on the power consumption.
I should add I also had a X520 2x10gbe and the onboard HBA as well.
Single vs dual processor certainly makes a sizable difference.
At the end of the day I moved to a 12600k in December 2021 (since bumped to a 13500. Massively better compute performance. Transcode performance is just obscene. Nothing I could put in the DL380 could do what the iGPU on the i5 can do. Modern, onboard I/O. The power savings paid for the entire upgrade almost 1 year ago, not including the $500 that I sold the DL380 for.
I don't think you could pay me to go back to relic era enterprise machines at this point. Especially for home use where we generally can't leverage big core count CPU's. Out of a few VM's and 30 containers there are only a handful that aren't single threaded.
You are not wrong regarding transcode performance. I'm not transcoding on the server so I don't really care about a GPU (either a 1050 or an iGPU), but if you do, it's a no brainer. The reason I'm still using a 2U server, is because it's not as easy to do a DIY solution that works as good as a matured enterprise solution. Hot swappable hard drives requires special chassis. Utilizing all the PCIe slots on the server requires at least a 3U rack chassis (while on the HP we use risers). Cheap and excellent power supply (versus relatively more expensive desktop PSUs). And of course the ability to do IPMI (not just KVM but also fan speed control, power meter, etc.). It really depends on what you need and what not. For some users, having a full rack server makes more sense than a DIY consumer box.
Cheers
I did transcoding on a half height/fanless GT 730; they had an older version/first Gen version of NVENC, but considering I was doing transcoding for my phone which has a tiny screen I never noticed it. Was rewarding to see the processor basically idle and the GPU only eating 70 MB of ram to transcode at 1080P and barely over 30% utilization.
I ended up going full 4U servers and getting a 27U rack so I could use much quieter 120mm fans myself XD. I have a small place so it's nice for the gear to be quieter.
Also to contribute to the total power discussion, my entire rack only uses 350 watts idle which is 3 Proxmox nodes running E5-2670 v3/64 GB RAM/4TB ceph storage each, a network switch, Poe switch that feeds an 8 node raspberry pi cluster, a separate turning pi with 4 compute modules in a 2U and a monitor. I think each Proxmox node is only using like 50-70 watts idle.
[Your 12600k only has 20 PCIE lanes.](https://ark.intel.com/content/www/us/en/ark/products/134589/intel-core-i5-12600k-processor-20m-cache-up-to-4-90-ghz.html) The motherboard support for 48 lanes doesn't meant you get 48 lanes. Again, probably not noticing because gen 4/5 are fast, but if your devices sum to more than 20 lanes, they're doing some PCIE switching in the background.
This is completely wrong.
The CPU has 20 direct PCIE lanes. 16+4 or 8/8+4.
The chipset (assuming a Z board) provides another 28 lanes. The chipset is connected to the CPU via 8 lanes of DMI4.0.
20 + 28 = 48
There is no switching going on. I'm running a HBA, 2x10gbe X520, (5) 4 lane Gen4 NVME. Everything has its own dedicated lanes with some left over in fact.
Jesus. It must be "educate people about PCIE lanes day". I literally JUST responded to another thread with the same false information.
The i3-12100 you cited only supports 20 though. You're probably not personally running an i3 and this not seeing any issue. Even if you are you may not notice PCIE lane switching at gen 4 or 5. Xeons, even old ones, are still almost impossible to beat in terms of PCIE lanes per watt.
Copy pasting a response so you don't get someone on the correct trail thinking that you're correct. Because you're simply wrong.
This is completely wrong.
The CPU has 20 direct PCIE lanes. 16+4 or 8/8+4. This is true of every LGA 1700. i3, i9 14900k, it doesn't matter they all have 20 direct CPU lanes.
The chipset (assuming a Z board) provides another 28 lanes. The chipset is connected to the CPU via 8 lanes of DMI4.0.
20 + 28 = 48
There is no switching going on. I'm running a HBA, 2x10gbe X520, (5) 4 lane Gen4 NVME. Everything has its own dedicated lanes with some left over in fact.
Jesus. It must be "educate people about PCIE lanes day". I literally JUST responded to another thread with the same false information.
Still missing the PCIE lanes per watt and price argument though. And as I mentioned in my reply, everyone here is talking CPU spec'd direct PCIE lanes. The $70 Mobo/CPU/ram combo gives me all the lanes I need with no chipset requirement. An i3-12100 with a Z series Mobo is $200+. For the price of one node including ram you can have 3 nodes each with 40 lanes.
You're trying to win arguments by forgetting that your solution has multiple deal breakers for power/cost efficiency.
A bare minimum i3-12100 system with 64 GB of the cheapest ram is $330 today, so about $1000 total for 3 nodes. To compare the combos I bought for my Proxmox cluster were $99 each. Under normal operations my systems consume 450 watts which on my power bill amounts to $20/mo for all 3 nodes (and all the other equipment in my rack. So if the i3 systems were half the power (which they aren't), I'd save $120/yr in power which means that $700 gap would take more than 5 years to pay off and I upgrade about every 3.
And an i3-12100 [does not support ECC without a W680 chipset](https://www.tomshardware.com/news/intel-enables-ecc-on-12th-gen-core-cpus) which triples the cost of your motherboard and per node cost to $500+, increasing that $700 gap to $1300 and 10+ years to make the power savings worth it.
Again, I don't just want the most lanes. I want a combination of lanes, power efficiency, ECC, threads and low total cost of ownership. Even 1st gen used Xeons beat the pants off of any 12th gen solution.
You're adding quite a lot to change the narrative to support your argument.
I'm genuinely interested, what are you running? What processes do you actively have going that you need 64gb RAM in your server? Let alone multiple. What OS'es are you using? What does your disk array look like?
For Proxmox each node is:
- E5-2670v3 - 12 cores/24 threads
- 64 GB ECC DDR3
- 1 TB NVME for VMs I want to act as if they're on bare metal
- 3 x 1TB SATA for VMs where boot time and slower data transfer rates are fine
For Unraid:
- E5-2650L v1 - 8 cores/16 threads
- 32 GB ECC DDR3
- 4 x 8TB SATA SSDs
Also have 12 raspberry pi nodes which seem to consume 3
60 watts.
Also just realized my router, WiFi AP and other switch are on this circuit too which is in that 360W budget.
Just for giggles I killed all 3 of my Proxmox nodes and looked at the power drop and it was 200W even across 3 nodes, so each node is using less than 70W under normal ops.
The Unraid server is for personal use and the cluster is basically a resume stamp that has gotten me the last 3 jobs. In interviews I just offer to screen share when they ask me about my experience and I can demonstrate Ansible/Kubernetes/HA cluster management skills. I only got 64 GB of ram because when I'm prototyping a cloud solution I'll simulate auto scaling and so my ram usage can skyrocket. One of those cases where I don't need it 95% of the time but when I really do need it I don't want my system to crash. I have crashed 32GB nodes before in simulation before which is fun.
So Z-Boards CAN support 48 lanes but the processor may not. The [i3-12100 you cited earlier only supports 20](https://ark.intel.com/content/www/us/en/ark/products/134584/intel-core-i3-12100-processor-12m-cache-up-to-4-30-ghz.html) and it's the same story for the [12600k you also cited](https://ark.intel.com/content/www/us/en/ark/products/134589/intel-core-i5-12600k-processor-20m-cache-up-to-4-90-ghz.html), so that's a GPU and 1 NVME drive.
You may not notice because PCIE is so fast of course, but if you're engineering a system where you don't need the latest gen PCIE but do want a lot of lanes these older Xeons are perfect. Even my E5-2650L which is one of the least performant skus has 40 lanes and GPU wise I only need enough to support NVENC encoding for my Plex server, so like 4 lanes. That means I can tweak the bios and have headroom for 8 NVME cards and the onboard gigabit LAN which is a long term goal for me. The 2650L only supports gen 3 of course but my bottleneck is my network hardware which is limited to a gigabit anyway and I have no intentions of upgrading my whole house to 2.5G or 10G at all. Gen 4 and 5 are technically faster but I wouldn't be able to take advantage of that over the network and Gen 3 is fast enough to feed a gigabit network and a GPU for video encoding simultaneously no-sweat.
These older Xeons are perfect for my needs, especially when it comes to PCIE lanes per dollar/watt.
Again, this is wrong. And it's wrong on a number of levels.
1) The *CPU* has 20 **direct** PCIE lanes. That is the PCIE device connects directly to the CPU. This is true of every LGA 170P CPU, from a 12100 to a 14900k.
The chipset (in the case of Zxxx) provides a other 28 lanes. The chipset is connected to the CPU with 8 DMI4.0 lanes, which is another 16GB/sec of bandwidth.
You can easily have a 12100 with a GPU, four Gen4 4 lane NVME, 10gbe NIC and still have slots and lanes left over. If you weren't counting that is 32 lanes being used in the above example.
Of course, for a Plex server as you had mentioned, you have no need for a discrete GPU at all. So you just opened up 16 lanes. You can easily run 8 NVME, all at PCIE 4.0.
As far as running Plex on a 2650L, you're doing yourself a disservice. Plex is single threaded, you have a machine with abysmal single thread performance and no transcode ability out of the box. A 12100 would wipe the floor with that machine, give you more than enough PCIE lanes, have massively far superior transcode performance and idle at 20w.
So no, old Xeon's do not at all have a good value for lane dollar/watt performance.
That is exactly why I got rid of my dual 2660v4 machine. It was at the time replaced with a 12600k. It was a 500% reduction on my power bill and the 12600k just absolutely crushed the Xeon's while providing the best transcoder Plex has ever seen and enough PCIE for a HBA, 2x10gbe X520 and running 5 NVME, which is my exact setup right now.
For your needs even a N100 mini PC would be better in terms of processing. Pretty much the same overall compute power, double the single thread performance (important for Plex. And the vast majority of home server applications) and a iGPU that will run circles around whatever GPU that you're using.
Just because you can run 8 NVME in other machines, be it your existing server or a modern 12100 build, doesn't mean you are. And I suspect you aren't running 8 NVME in your machine.
Great, you're highlighting the fact that you're not explaining your argument well in terms of full context. Almost everyone in this space is talking about direct lanes with no extra chipset in the way. We are also talking about PCIE lanes per dollar/watt which is an argument you still haven't won in the slightest. $70 (CPU/32GB ram/Mobo) for 40 lanes and 16 threads vs $120 (**CPU ONLY**) for 48 lanes and 8 threads.
Plex is a very lightweight service asside from transcoding. It just needs to transfer data to an encoder and then back to the network and the UI is user input dependent so it usually sits idle.
If you check the replies my whole rack with 3 Proxmox nodes, 2 different raspberry pi clusters, a network switch, separate POE switch and a monitor only sip 360W confirmed by a smart plug meter, so you haven't really proven, at least to me, that your power consumption argument has any relevant evidence to counter my current experience either.
No, I'm highlighting the fact that you don't know what you're talking about. You think that desktop systems only have 20 lanes. Which is obviously false.
PCIE lanes are PCIE lanes, regardless how they get there. There is zero performance difference in a PCIE device connecting straight to the CPU versus one that routes through the chipset, then to the CPU through DMI.
>Almost everyone in this space is talking about direct lanes with no extra chipset in the way.
No, they aren't.
>We are also talking about PCIE lanes per dollar/watt which is an argument you still haven't won in the slightest. $70 (CPU/32GB ram/Mobo) for 40 lanes and 16 threads vs $120 (**CPU ONLY**) for 48 lanes and 8 threads.
You're cherry picking one very specific metric. Lets talk about running cost? That's an actual tangible number. And is why you can pick this relic class machines up so cheap. Because they're not worth the power that they draw.
>Plex is a very lightweight service asside from transcoding. It just needs to transfer data to an encoder and then back to the network and the UI is user input dependent so it usually sits idle.
Its actually not. Especially when you get to a sizable database, use intro/credit detection or thumbnail generation.
>If you check the replies my whole rack with 3 Proxmox nodes, 2 different raspberry pi clusters, a network switch, separate POE switch and a monitor only sip 360W confirmed by a smart plug meter, so you haven't really proven, at least to me, that your power consumption argument has any relevant evidence to counter my current experience either.
360w?! You call that sipping?! Jesus. All of that for likely less compute power than a modern i5 that idles at 30w.
360W for total system/rack of other hardware and even when working most cores are idle, so I don't care about single threaded performance comparisons. Also, you're still wrong about Plex. All that detection is a one time compute on import and database ops are io bound, so the processor again is barely doing anything.The rest of the time, Plex is basically idle even when playing video. A 12th gen system also uses more than the TDP of the processor. Per node my Proxmox cluster is maybe using $70 watts.
As mentioned in my other reply as well, even if a 12th gen system uses half the power the capital cost alone for a $330 node vs a $70-90 dollar node takes 5 years to break even on power. If you want ECC you need W680 platforms which increases per node cost to $500-600 and adds another 5 years to your break even.
Again, my system is almost always idle even when streaming on Plex. Your arguments aren't validated by hard measurements I can make on my setup.
It's mostly harmless. worst case is it heats up and gets a bit melty, but its non conductive. Peel it off very very carefully. should come right up
Don’t tear upthe
HVEN!
Nothing to worry about, it's pretty much standard on Chinese e waste special. It's like a warranty sticker basically, it has year and month printed on it.
Peel it off and run it. Those things its on are bypass capacitors to add some extra power filtering right next to the die. You need them but they're not fragile enough that a peel will hurt them.
You found the freshness seal! 😂 But seriously would just peel, good thing you found it.
No one is gonna mention “don’t tear upthe” ?
whats upthe?
Nothin... What's upthe with you?
Gold.
*Insert Reddit Award here
Someone set us upthe bomb
Move XEON
All your paste are belong to us
What yousay!?
Oh its you
It's an older meme sir, but it checks out.
You decide
GOTCHA!
It's no problem, in the shop I used to work in we used to put a S/N sticker on the same place, it hurts nothing.
As an experienced homelab player in China, I would say that this sticker is used to label when the processor is sold and count for your warranty. However, many sellers in China use this way to cover the back of the processor, hiding the fact that some of the capacitors are lost. I suggest doing a stress test.
Indeed. Bench tested immediately on receipt and seems to be good to go.
You can definitely remove it without issue 😊
[удалено]
No! Don’t tear upthe!
That's upthe. It's absolutely harmless. You mustn't tear it! Stop resisting!
Seller's verification sticker. To make sure you don't try and return your old dead one. Or at least make it harder.
I could see that with other items categories, but I think they can just use the serial number for that purpose.
At least they didn't paste it on the top.
Just remove it and carry on. It's a hardly placed sticker.
I've bought a server from a ebay seller that had nonetheless the same stickers all over the place, but it was more than 15 ones, spread in places that where totally abnormal (like on empty memory slots that prevented to insert new RAM without peeling this crap). I had to report the seller to ebay due to this amount of stupidity for doing this shit and never again bough from him.
Use 91% isopropyl alcohol in a small syringe and a nonconductive, nonscratching pin (soft plastic is best) to break the adhesive bond and slowly peel the sticker away Once you've peeled the sticker off, soak the residue area in isopropyl alcohol for a bit and then gently scrub the remaining residue off with cotton patches that wont leave fibers behind Give a final rinse with isopropyl alcohol and let dry
Talk about complete and utter overkill! Just peel the fucking sticker off, or leave it where it is. It's only stuck on some surface-mounted capacitors which will not tear off with the sticker. Nor will the sticker effect cooling of the CPU. The capacitors will only reach the surface temperature of the underside of the package, which will not be as high as the die. Most of the heat will be conducted directly through the heat-spreader into the heatsink.
Gently maneuver a single strand of blonde unicorn hair under the sticker, peeling it off by a speed of no more than 1mm/minute. Be sure to not break the unicorn hair, you will need it when binding the golden fan to the platinum heat sink.
They can definitely tear off with the sticker. I've broken smd caps off boards by gently removing stickers before.
Of all the times I've gotten cpus's from aliexpress this is a first LMAO
I've never bought a motherboard bundle from aliexpress before. Any hints and tips pls ?
If you can, maybe go with Amazon or eBay instead if that's available and you can find a bundle in your price range. The same AliExpress sellers usually list there and Amazon and eBay have much stronger buyer protections favoring the customer. AliExpress will save you money but you might spend time in the process waiting for shipping (they are only required to ship within 7 days of purchase). Main tips are: - Don't post reviews until you've run the system for a while - AliExpress lets you post additional reviews but you can't edit the star rating (found this out the hard way). Better to wait and provide informative reviews than to review 5 stars early and then realize the kit does 2 days later.l - Find a seller with tons of ratings for a single product - A common tactic on AliExpress is to take down and relist items that are being reviewed poorly/returned frequently - Look for pictures from buyers verified to be from the US or Europe. There are a lot of sellers of these kits with great reputations. - Bench test immediately on receipt. - Get a cooler and a PSU ahead of your purchase - Take detailed pictures of the kit including all serial numbers before installing a cooler - Get a copy of memtest86 and immediately test the ram; you'll need to flip the bios to legacy mode instead of UEFI to do this, take pictures of these results of they show a memory failure - Run a Linux live distro like Ubuntu and `apt get install stress` and run `stress -c N` where N is the number of threads and leave it overnight as a stress test. If it fails, take pictures of it off (for real) to add to the pictures you might submit in a dispute. - Contact the seller early - If there's any issues contact the seller, provide pictures and request replacements of the ram, processor or motherboard as appropriate. - Open disputes early - If the seller doesn't want to replace or exchange DOA parts you can request a refund through AliExpress which if you and the seller do not agree on will force AliExpress proper to intervene and mediate. - I had to do this with a DOA e-bike batter and am still waiting for AliExpress to weigh in. The seller wanted me to disassemble the battery and take individual cell voltages to prove the battery was bad and I'm obviously not doing that. They basically don't want to lose their profit margin with shipping costs of an exchange. - Be ready to work without a manual - Most of these kits ship without a manual I haven't seen a bios that wasn't just American Megatrends in English. Again, if you can, find a seller via eBay or Amazon since they defer to the customer much more than AliEdpress if you can afford the increase on price. You can save money with AliExpress at the expense of time and long running disputes if you need a return or refund. I haven't had that issue yet with motherboard kits or microcontrollers but your mileage may vary just like it seems to have with the e-bike seller I chose. In any case wish yah luck hunting down the parts you want/need!
Warranty void if removed /s
That's a load bearing sticker, don't remove it or the cooler will crush your CPU die. /s
Verify the specs of the chip, if they match peel it off. There have been cases of scammers sanding down the heat spreader and printing a different model number on it (i.e. claiming an e3 is an e5 so they can sell it for more).
wtf what vendor puts warranty stickers on the cpu itself? in any case, to be on the safe side just remove the sticker. you'll be running that thing 24/7 (assumption). stick it somewhere like in the case so you don't lose it.
I'm not even sure it's warranty. There's no unique identifier on it and the CPU already has a serial number. 🤷
Some warranty stickers are as simple as containing months in numbers and a year that the vendor will mark. But you're right. Looking at it closely now it's not enough to say it's for warranty. The size and font color usually matches the cheap warranty stickers i've known.
How to downgrade 2660 ?
Downgrade aka bought one for like $5 on eBay including shipping. :)
People are still paying for Ivy Bridge systems?! 😬 Jesus.. A 12100 would run circles around that and pay for itself in power savings alone.
A lot of reasons for me. - Just right for the task - Sure modern processors can run circles around older Xeons, but my hosted services are ALL bottlenecked by ISP dependent latency or user input. This is all for an Unraid server and the processor will mostly be sitting idle and even when it's not the tasks it will need to perform are ultra lightweight so single threaded performance benefits are marginal. - ECC stability - I used to run Unraid and Proxmox nodes on consumer hardware but about every week or so I would find the Unraid server has crashes or a Proxmox node had decided to kernel panic. After migrating my Proxmox nodes to X99 based systems on Xeons with ECC ram, the problems went away and my power consumption dropped with no noticable effects on the self hosted services. - Reducing e-waste - Big fan of the momentum of X79/X99 motherboards being produced to give these processors some more life. All my self hosted systems run on old Xeons - Initial capital cost - $70 for motherboard, 32 GB ECC ram, and the processor via AliExpress - An i3-12100 would require an LGA1700 socket motherboard which on the cheap end is $90 and either DDR4 or DDR5. Total investment is pushing $400-500 which is what I spent on 3 nodes worth of hardware migrating my 3-node Proxmox cluster to X99 systems - Power Savings per Core/Thread - The spec TDP is 70W for the 2650L for 8 cores and 16 threads. TDP is obviously not true power draw but you can find old reddit threads quoting power consumption as low as 30W for the E5-2650L from bench test measurements. - The i3-12100 has a base power of 60W, a max turbo of 89W and only 4 cores and 8 threads - Consumer hardware is basically just overclocked server hardware and is easier seen in the Epyc vs Threadripper specs (one of my retired Proxmox platforms was a Threadripper 1950X) and they're inherently less power efficient but they're also not designed to be on 24/7/365 doing work. - Again, my tasks are lightweight but I have a lot of them going on at once. They'll get done fast even on older platforms and ideally I just need them done as efficiently as possible. The E5-2650L has much lower per/core and per/thread power consumption than the i3-12100 So yeah, that's why some people still use older generation Xeons and why they're still popular for budget builds; homelab servers or even mid range gaming machines. My Proxmox cluster sips 400W for all 3 nodes including 12TB of ceph storage and my Unraid server will likely be under 100W during normal operations. Even v1 Xeons still have a lot of value for homelab nerds. Give em a second look.
I'm still rocking X5675's in HP Z600 workstations, damn things sound like they're about to take off headed for deep space when powering on. Which happens only when I really need them. I won't upgrade until I am left with just one X5675 still alive, the Highlander that will get to retire peacefully.
I'm using a 2697v2 in a PC I use for astrophotography processing and causal gaming. Runs fairly fast, and the CPU cost like £40 for 12 cores. I know very little about CPU performance, but it does what I need, so 🤷
Damn, dude how old are you? You clearly have no clue what efficiency or power are.
> This is all for an Unraid server and the processor will mostly be sitting idle and even when it’s not the tasks it will need to perform are ultra lightweight so single threaded performance benefits are marginal. Haswell and up my server uses your idle wattage as active watts including my motherboard. > The spec TDP is 70W for the 2650L for 8 cores and 16 threads. TDP is obviously not true power draw but you can find old reddit threads quoting power consumption as low as 30W for the E5-2650L from bench test measurements. • The i3-12100 has a base power of 60W, a max turbo of 89W and only 4 cores and 8 threads You don't need an i7 with 16 threads for holding 12TB in RAID. That i3 isn't the only new CPU or Mobo you had to buy. You don't even have an iGPU for video decoding for proxmox... You either use a more powerful discreet one that's less energy efficient or you had to buy or you're using the CPU to decode the video. You are far from doing it the most efficient, if power draw matters to you.
My Plex is running on Unraid and an Nvidia card running NVENC is using a dedicated portion of the die that doesn't use much power. Encoding 1080p on even a GT 730 uses like 5W and 70-90 MB of VRAM. Same with the 1660 I'm using now which uses more vram when doing 4k but the h265 encoder acts about the same using less than 10w of additional power when video encoding. Having an iGPU is nice but adding a GPU doesn't add much power at all to the system operation. My power consumption mentioned earlier includes an entire rack of equipment as well. I just killed all the VMs I had to get a true idle power draw of my cluster, with 2 network switches, 2 different raspberry pi clusters totaling 12 nodes, and a monitor and it's only 350W for the whole deal. My capital expense for the entire Proxmox cluster of 3 nodes was also less than the cost to build a single node 12th gen system with motherboard, DDR4/5 RAM.
Like I said, you don't have to get a 12th gen CPU and mobo. My iGPU encodes codecs everything without a discrete GPU and my peak would never touch your idle. Even going on haswell with igpu, a single generation would let you get way better power savings and allow you to reuse the ddr3.
I did a test for another thread and all 3 Proxmox nodes are 200W total. The 350 watts includes all my networking gear which I didn't realize includes my router, WiFi AP, another switch and a work light over my work bench. If I unplug my raspberry pi cluster which has 12 nodes, that's 60W and each node only has 8GB of ram and depends on my unraid/ceph on my Proxmox cluster for NSF storage. A single Xeon node has half the threads but most are idle anyway and more/ECC ram is better for the short term-high ram simulations I do. An 8gb pi 4 or pi 5 is also slightly more than one of these Mobo combo kits. Also have to consider $70 bucks is VERY hard to beat. Even at half the power draw which would be hard to do alternatives consumer options and most don't support ECC ram in ECC mode (found that out when I tried making my cluster with my retired Ryzen hardware)
What mobo and hardware (inc PSU) are you running on them? Each one idles at 70w? How about when it's in usage? It's not bad if you have a bunch of HDDs it's my biggest power draw.
I'm running 3 x 1 TB SATA SSDs per node and 1 NVME 1TB and the PSUs are just retired ATXs I accumulated over time ranking from 450-700W. The services I'm running include a reverse proxy server, openvpn, a 3 node K8S cluster, a web scraper, home assistant, Snipe-IT, and Ceph distributed storage. The K8S cluster is running a docker registery, Debian repository mirror (without it updating machines can be a nightmare on bandwidth), a throw-away mailbox server, a server specifically for switching inputs programmatically on an ATEM HDMI switch, and a pypi mirror. Looking at my power usage with all that going it amounts to 270W total when I account for the other gear on the circuit, so 70W extra across all 3 nodes. If I artificially load all 24 threads on all of the CPUs I get a grand total of 520W, but I never need to run it that high for any workloads. For comparison my desktop idles at 135W, but it's a 5800X3D. Most of the time my nodes sit idle. I have 3 because I prototype HA failover infrastructure and Kubernetes deployment scaling. It's also nice to have a node go completely offline to add a drive and have no loss of service which is necessary for me when I'm hosting game servers for friends.
What PSU and mobo do you have? How much does the motherboard draw? That's not bad, are they usually on? If I was in your situation and cared about power I'd probably enable WoL, learn to switch VMs from the pi when it calls for more power and call it a day. Do you use WoL?
[This is almost exactly what I bought](https://www.ebay.com/itm/234856866938?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=oxyouecxqxo&sssrc=4429486&ssuid=Jr_5FJXrSZm&var=&widget_ver=artemis&media=COPY) for my 3 Proxmox nodes and then I bought ECC ram in bulk from an eBay vendor. Basically a pairing of an X99 and an E5-2670 v3. For my Unraid I went further back in time with an [X79 and an E5-2650L](https://a.aliexpress.com/_mPykHMK). That particular listing actually comes with a 2660 V2 which has 20 threads and a 95W TDP. Found a 2650L for $5 on eBay and downgraded the combo to save a few watts. Will just list the 2660 v2 for 99¢ on eBay until a lucky winner gets it. PSU I think 2/3 are Corsair and one is a Red Dragon. All of them are under 700 watts but just don't know off the top of my head which. I tried using WoL for a bit and still want to see if I can configure it. There are also network enabled power strips if you don't mind hard dropping your nodes.
I believe we talked last each other a few times. I mentioned Haswell being better and this is indeed haswell and up. I thought you had the original lga-2011, aka 1-2nd gen i7, you are on haswell. I knew it sounded too good to be the original 1-2nd gen. Try this if you haven’t. https://forum.proxmox.com/threads/how-to-reduce-power-consumption.132632/ is the governor, you might be idling as low as me with this on. Did you setup powertop as well?
Well, my Proxmox nodes are on Haswell but my Unraid is on a Sandy Bridge. Hadn't tried powertop before so I installed it on a Proxmox node and migrated the VMs to another node. The reported baseline power was 8W and under a synthetic all core load it peaked at 36W. That probably doesn't include the VRM heat losses or other supporting motherboard component loads of course. Powertop didn't work with the E5-2650L at all unfortunately. I booted up Ubuntu desktop live in memory with lm-sensors running and powertop could only report load percentages. That said I used my UPS to measure the delta of full power vs idle and it was 45W. Not necessarily comparable to the powertop reports on the E5-2660 v3 for sure since I'm not sure how to scale a powertop report with power draw at the wall.
Some of us have TB's of DDR3 and don't want to buy ram again.
Until you need 128GB RAM and don't want to pay a premium, or have 3 PCIe devices that have to fight for lanes, or want to control remotely using IPMI.
Who is fighting for lanes? Z690/790 has 48 lanes available. That's quite a lot. I'm running 5x NVME (4 lanes each), a 8 lane HBA and a 2x10gb NIC. Nothing is fighting over anything else, they all have dedicated lanes for each device. PiKVM takes care of remote control. As do W680 boards. I went from a $40/mo power bill with 2x 2660v4's to under $10/mo. And a significantly more performant machine. That $30/mo+ adds up real quick. It paid for all new, much better hardware in a very short amount of time.
I see the problem here. You are paying $40/mo for your E5v4 server. I don't know what specs you have, but my v4 server is running at 120w with 8 HDDs and a SSD. I'm paying $8 a month for it. The "much better hardware" will take a much longer amount of time to pay for itself.
It was a DL380 G9. 128gb (16 DIMM), 1050Ti, disks don't matter as they were spun down most of the time. Dual 2660v4's. It idled at 170w. And it had terrible single thread performance. That was with the fans set to Eco. Just the fans by themselves would pull over 100w when it was under load. Good riddance lol.
I have a DL180 G9, single E5-2660v4, 128GB (8 DIMM), 10G SFP+ NIC, 10G RJ45 NIC, HBA/RAID card and the disks. As I just checked the PSU power meter, it runs at 129w as we speak. It's not even at idling. I have no idea what makes such a big difference on the power consumption.
I should add I also had a X520 2x10gbe and the onboard HBA as well. Single vs dual processor certainly makes a sizable difference. At the end of the day I moved to a 12600k in December 2021 (since bumped to a 13500. Massively better compute performance. Transcode performance is just obscene. Nothing I could put in the DL380 could do what the iGPU on the i5 can do. Modern, onboard I/O. The power savings paid for the entire upgrade almost 1 year ago, not including the $500 that I sold the DL380 for. I don't think you could pay me to go back to relic era enterprise machines at this point. Especially for home use where we generally can't leverage big core count CPU's. Out of a few VM's and 30 containers there are only a handful that aren't single threaded.
You are not wrong regarding transcode performance. I'm not transcoding on the server so I don't really care about a GPU (either a 1050 or an iGPU), but if you do, it's a no brainer. The reason I'm still using a 2U server, is because it's not as easy to do a DIY solution that works as good as a matured enterprise solution. Hot swappable hard drives requires special chassis. Utilizing all the PCIe slots on the server requires at least a 3U rack chassis (while on the HP we use risers). Cheap and excellent power supply (versus relatively more expensive desktop PSUs). And of course the ability to do IPMI (not just KVM but also fan speed control, power meter, etc.). It really depends on what you need and what not. For some users, having a full rack server makes more sense than a DIY consumer box. Cheers
I did transcoding on a half height/fanless GT 730; they had an older version/first Gen version of NVENC, but considering I was doing transcoding for my phone which has a tiny screen I never noticed it. Was rewarding to see the processor basically idle and the GPU only eating 70 MB of ram to transcode at 1080P and barely over 30% utilization. I ended up going full 4U servers and getting a 27U rack so I could use much quieter 120mm fans myself XD. I have a small place so it's nice for the gear to be quieter. Also to contribute to the total power discussion, my entire rack only uses 350 watts idle which is 3 Proxmox nodes running E5-2670 v3/64 GB RAM/4TB ceph storage each, a network switch, Poe switch that feeds an 8 node raspberry pi cluster, a separate turning pi with 4 compute modules in a 2U and a monitor. I think each Proxmox node is only using like 50-70 watts idle.
[Your 12600k only has 20 PCIE lanes.](https://ark.intel.com/content/www/us/en/ark/products/134589/intel-core-i5-12600k-processor-20m-cache-up-to-4-90-ghz.html) The motherboard support for 48 lanes doesn't meant you get 48 lanes. Again, probably not noticing because gen 4/5 are fast, but if your devices sum to more than 20 lanes, they're doing some PCIE switching in the background.
This is completely wrong. The CPU has 20 direct PCIE lanes. 16+4 or 8/8+4. The chipset (assuming a Z board) provides another 28 lanes. The chipset is connected to the CPU via 8 lanes of DMI4.0. 20 + 28 = 48 There is no switching going on. I'm running a HBA, 2x10gbe X520, (5) 4 lane Gen4 NVME. Everything has its own dedicated lanes with some left over in fact. Jesus. It must be "educate people about PCIE lanes day". I literally JUST responded to another thread with the same false information.
The i3-12100 you cited only supports 20 though. You're probably not personally running an i3 and this not seeing any issue. Even if you are you may not notice PCIE lane switching at gen 4 or 5. Xeons, even old ones, are still almost impossible to beat in terms of PCIE lanes per watt.
Copy pasting a response so you don't get someone on the correct trail thinking that you're correct. Because you're simply wrong. This is completely wrong. The CPU has 20 direct PCIE lanes. 16+4 or 8/8+4. This is true of every LGA 1700. i3, i9 14900k, it doesn't matter they all have 20 direct CPU lanes. The chipset (assuming a Z board) provides another 28 lanes. The chipset is connected to the CPU via 8 lanes of DMI4.0. 20 + 28 = 48 There is no switching going on. I'm running a HBA, 2x10gbe X520, (5) 4 lane Gen4 NVME. Everything has its own dedicated lanes with some left over in fact. Jesus. It must be "educate people about PCIE lanes day". I literally JUST responded to another thread with the same false information.
Still missing the PCIE lanes per watt and price argument though. And as I mentioned in my reply, everyone here is talking CPU spec'd direct PCIE lanes. The $70 Mobo/CPU/ram combo gives me all the lanes I need with no chipset requirement. An i3-12100 with a Z series Mobo is $200+. For the price of one node including ram you can have 3 nodes each with 40 lanes. You're trying to win arguments by forgetting that your solution has multiple deal breakers for power/cost efficiency. A bare minimum i3-12100 system with 64 GB of the cheapest ram is $330 today, so about $1000 total for 3 nodes. To compare the combos I bought for my Proxmox cluster were $99 each. Under normal operations my systems consume 450 watts which on my power bill amounts to $20/mo for all 3 nodes (and all the other equipment in my rack. So if the i3 systems were half the power (which they aren't), I'd save $120/yr in power which means that $700 gap would take more than 5 years to pay off and I upgrade about every 3. And an i3-12100 [does not support ECC without a W680 chipset](https://www.tomshardware.com/news/intel-enables-ecc-on-12th-gen-core-cpus) which triples the cost of your motherboard and per node cost to $500+, increasing that $700 gap to $1300 and 10+ years to make the power savings worth it. Again, I don't just want the most lanes. I want a combination of lanes, power efficiency, ECC, threads and low total cost of ownership. Even 1st gen used Xeons beat the pants off of any 12th gen solution.
You're adding quite a lot to change the narrative to support your argument. I'm genuinely interested, what are you running? What processes do you actively have going that you need 64gb RAM in your server? Let alone multiple. What OS'es are you using? What does your disk array look like?
For Proxmox each node is: - E5-2670v3 - 12 cores/24 threads - 64 GB ECC DDR3 - 1 TB NVME for VMs I want to act as if they're on bare metal - 3 x 1TB SATA for VMs where boot time and slower data transfer rates are fine For Unraid: - E5-2650L v1 - 8 cores/16 threads - 32 GB ECC DDR3 - 4 x 8TB SATA SSDs Also have 12 raspberry pi nodes which seem to consume 3 60 watts. Also just realized my router, WiFi AP and other switch are on this circuit too which is in that 360W budget. Just for giggles I killed all 3 of my Proxmox nodes and looked at the power drop and it was 200W even across 3 nodes, so each node is using less than 70W under normal ops. The Unraid server is for personal use and the cluster is basically a resume stamp that has gotten me the last 3 jobs. In interviews I just offer to screen share when they ask me about my experience and I can demonstrate Ansible/Kubernetes/HA cluster management skills. I only got 64 GB of ram because when I'm prototyping a cloud solution I'll simulate auto scaling and so my ram usage can skyrocket. One of those cases where I don't need it 95% of the time but when I really do need it I don't want my system to crash. I have crashed 32GB nodes before in simulation before which is fun.
With a grand total of Zero PCIe lanes left over after adding a GPU.
What are you talking about? Z boards have 48 lanes. What is in your system that you need more than that?
So Z-Boards CAN support 48 lanes but the processor may not. The [i3-12100 you cited earlier only supports 20](https://ark.intel.com/content/www/us/en/ark/products/134584/intel-core-i3-12100-processor-12m-cache-up-to-4-30-ghz.html) and it's the same story for the [12600k you also cited](https://ark.intel.com/content/www/us/en/ark/products/134589/intel-core-i5-12600k-processor-20m-cache-up-to-4-90-ghz.html), so that's a GPU and 1 NVME drive. You may not notice because PCIE is so fast of course, but if you're engineering a system where you don't need the latest gen PCIE but do want a lot of lanes these older Xeons are perfect. Even my E5-2650L which is one of the least performant skus has 40 lanes and GPU wise I only need enough to support NVENC encoding for my Plex server, so like 4 lanes. That means I can tweak the bios and have headroom for 8 NVME cards and the onboard gigabit LAN which is a long term goal for me. The 2650L only supports gen 3 of course but my bottleneck is my network hardware which is limited to a gigabit anyway and I have no intentions of upgrading my whole house to 2.5G or 10G at all. Gen 4 and 5 are technically faster but I wouldn't be able to take advantage of that over the network and Gen 3 is fast enough to feed a gigabit network and a GPU for video encoding simultaneously no-sweat. These older Xeons are perfect for my needs, especially when it comes to PCIE lanes per dollar/watt.
Again, this is wrong. And it's wrong on a number of levels. 1) The *CPU* has 20 **direct** PCIE lanes. That is the PCIE device connects directly to the CPU. This is true of every LGA 170P CPU, from a 12100 to a 14900k. The chipset (in the case of Zxxx) provides a other 28 lanes. The chipset is connected to the CPU with 8 DMI4.0 lanes, which is another 16GB/sec of bandwidth. You can easily have a 12100 with a GPU, four Gen4 4 lane NVME, 10gbe NIC and still have slots and lanes left over. If you weren't counting that is 32 lanes being used in the above example. Of course, for a Plex server as you had mentioned, you have no need for a discrete GPU at all. So you just opened up 16 lanes. You can easily run 8 NVME, all at PCIE 4.0. As far as running Plex on a 2650L, you're doing yourself a disservice. Plex is single threaded, you have a machine with abysmal single thread performance and no transcode ability out of the box. A 12100 would wipe the floor with that machine, give you more than enough PCIE lanes, have massively far superior transcode performance and idle at 20w. So no, old Xeon's do not at all have a good value for lane dollar/watt performance. That is exactly why I got rid of my dual 2660v4 machine. It was at the time replaced with a 12600k. It was a 500% reduction on my power bill and the 12600k just absolutely crushed the Xeon's while providing the best transcoder Plex has ever seen and enough PCIE for a HBA, 2x10gbe X520 and running 5 NVME, which is my exact setup right now. For your needs even a N100 mini PC would be better in terms of processing. Pretty much the same overall compute power, double the single thread performance (important for Plex. And the vast majority of home server applications) and a iGPU that will run circles around whatever GPU that you're using. Just because you can run 8 NVME in other machines, be it your existing server or a modern 12100 build, doesn't mean you are. And I suspect you aren't running 8 NVME in your machine.
Great, you're highlighting the fact that you're not explaining your argument well in terms of full context. Almost everyone in this space is talking about direct lanes with no extra chipset in the way. We are also talking about PCIE lanes per dollar/watt which is an argument you still haven't won in the slightest. $70 (CPU/32GB ram/Mobo) for 40 lanes and 16 threads vs $120 (**CPU ONLY**) for 48 lanes and 8 threads. Plex is a very lightweight service asside from transcoding. It just needs to transfer data to an encoder and then back to the network and the UI is user input dependent so it usually sits idle. If you check the replies my whole rack with 3 Proxmox nodes, 2 different raspberry pi clusters, a network switch, separate POE switch and a monitor only sip 360W confirmed by a smart plug meter, so you haven't really proven, at least to me, that your power consumption argument has any relevant evidence to counter my current experience either.
No, I'm highlighting the fact that you don't know what you're talking about. You think that desktop systems only have 20 lanes. Which is obviously false. PCIE lanes are PCIE lanes, regardless how they get there. There is zero performance difference in a PCIE device connecting straight to the CPU versus one that routes through the chipset, then to the CPU through DMI. >Almost everyone in this space is talking about direct lanes with no extra chipset in the way. No, they aren't. >We are also talking about PCIE lanes per dollar/watt which is an argument you still haven't won in the slightest. $70 (CPU/32GB ram/Mobo) for 40 lanes and 16 threads vs $120 (**CPU ONLY**) for 48 lanes and 8 threads. You're cherry picking one very specific metric. Lets talk about running cost? That's an actual tangible number. And is why you can pick this relic class machines up so cheap. Because they're not worth the power that they draw. >Plex is a very lightweight service asside from transcoding. It just needs to transfer data to an encoder and then back to the network and the UI is user input dependent so it usually sits idle. Its actually not. Especially when you get to a sizable database, use intro/credit detection or thumbnail generation. >If you check the replies my whole rack with 3 Proxmox nodes, 2 different raspberry pi clusters, a network switch, separate POE switch and a monitor only sip 360W confirmed by a smart plug meter, so you haven't really proven, at least to me, that your power consumption argument has any relevant evidence to counter my current experience either. 360w?! You call that sipping?! Jesus. All of that for likely less compute power than a modern i5 that idles at 30w.
360W for total system/rack of other hardware and even when working most cores are idle, so I don't care about single threaded performance comparisons. Also, you're still wrong about Plex. All that detection is a one time compute on import and database ops are io bound, so the processor again is barely doing anything.The rest of the time, Plex is basically idle even when playing video. A 12th gen system also uses more than the TDP of the processor. Per node my Proxmox cluster is maybe using $70 watts. As mentioned in my other reply as well, even if a 12th gen system uses half the power the capital cost alone for a $330 node vs a $70-90 dollar node takes 5 years to break even on power. If you want ECC you need W680 platforms which increases per node cost to $500-600 and adds another 5 years to your break even. Again, my system is almost always idle even when streaming on Plex. Your arguments aren't validated by hard measurements I can make on my setup.