This was apparently a Linde installation custom built for TSMC in Arizona.[1]
Nitrogen, oxygen, and argon are extracted from air on-site and purified.
That's Linde's primary business; liquefying and distilling air. This isn't some little local company or a company operating outside their area of expertise.
Those gases are storeable, so it's surprising there wasn't enough tank capacity to deal with outages.
The site plan [2] shows "Gas Plant 1", and future "Gas Plant 2" and "Gas Plant 3". The gas plants are across a small road from the fab and feed the plant directly. Once Gas Plants 2 and 3 were built, there would be redundancy, but at this stage, there isn't a backup. The plan doesn't show a large tank farm, so they can't store gases in bulk.
Unless things have changed a lot since I fled semiconductor manufacturing, you would still need silane tanks at least. I’m as surprised as you that they don’t have buffer tanks.
I don't think Linde™ needs an add, everyone knows Linde™ is the most reliable partner in producing and storing gases in all available purity classes.
(joke off, it's probably not an add, but they were excited to share the reason you see Linde on all sorts of gas tanks all over the place. It's actually quite common and if you see it once you see it everywhere. )
Possibly but also plausible is that its a deep joke that everyone is in on.
When googling the company, the marketing slogan that comes up is "Linde is Everywhere" but that works on so many levels. They sell air, air is everywhere. Therefore Linde is everywhere.
They are a company that sells air: that stuff that people breathe. Forget this AI nonsense. Jensen has to constantly pull something out of rear to keep food on the table. These guys sell air. What a business. :)
This isn’t very big news. Issues occur during bring-up often. Linde’s processes are possibly so power intensive that failing over to generator power is not possible. TSMC is right to put Linde on notice since Linde should have a PFMEA and control plan to eliminate any root causes for downtime. I suspect in the long term TSMC has plans to insource this if the issue persists. Scrap happens sometimes during manufacturing, if the writer only has journalism experience and no manufacturing experience then they may not have a conceptual understanding of acceptable first pass yield. After all, the TSMC logo features failing parts!
In many ways I agree with you, but the problem statement (constrained/exhausted gas supply from vendor) makes it seems like this was not just line down, but the whole factory stopped for a few hours. Line down is a miserable migrane but still managable... while a whole factory stoppage makes a lobotomy seem like a good idea. It also sounds like there was not enough forewarning to park critical customer wafers in a "safe" stage of the process.
Even so, I also would still call this another monday at a semiconductor factory. Welcome! Here we play a nearly endless game of whack-a-mole. Here's your mallet and your towel. Now whack enough of the moles hard enough until they stop coming back (at least through the same holes). Beware the alpha moles.
By any road, I am surprised to see even this high-level perspective on a quality event disclosed to the mainstream public; I thought this was not standard practice. I enjoyed the read.
> forcing the facility to shut down for at least a few hours
> As a result, the company had to scrap thousands of wafers
Anything involving wet chemistry, photoresist, furnaces, etc. is very time-constrained. You can't let wafers sit around indefinitely. Certain process steps must be followed up very quickly to avoid scrap.
This is why you dont see redundant power for manufacturing lines. A 3nm line needs hundreds of megawatts to operate. You cant clear queued lots without a fully functional line. There's not much you could save by keeping part of the line operational.
TSMC has backup generators in their AZ fab. You actually have to have backup power or a few hundred millisecond blip could cause days or weeks of tool down time. You should see what happens when you lose the ability to keep a clean room at temp/humidity/airflow...it's weeks or months.
It didn’t happen, but the facilities team at the fab where I worked was seriously considering installing a flywheel to cover power bumps. What I don’t get about this story is how this actually happened. All our process gasses were out in a tank farm and we knew how much pressure we had. We would have stopped the line if there wasn’t enough to proceed. Were they separating air onsite or something?
I was very impressed by the modest little fab I worked at having thousands of lead acid batteries for momentary takeover, and 8 five-megawatt locomotive engines for longer term redundancy. Apparently their steady state usage was 25MW, which allowed still having a hot spare and concurrent downtime for two of the locomotive generator units.
Yes, Linde has an onsite plant and is building two more.
For some processes, stopping will botch the wafer. In the event of a gas shortage, do plants plan which lines to take down first, and which lines should complete a process step?
The way this worked at the fab where I was, was that facilities would have paged everybody, and whoever needed to hold wafers would do so. You could mark your equipment down or unavailable for a particular step. I don’t know what we would have done if it was “hey, we lost dry nitrogen a minute ago.” I think at that point you lose a lot of wafers in wet cleans.
In the case of a power interruption at the fab, consequences were highly dependent on the equipment and the unit process. A prolonged power interruption to diffusion was the worst case scenario. You’d have 150 wafers in the furnace, and any significant deviation from the nominal temperature profile meant they were all scrap. Worse, if the furnace cooled off, you had to scrap the quartz boat the wafers rode in, too. Other processes had a smaller blast radius but were even more of a headache to disposition. Implant, you’d lose beam and probably lose vacuum too. Then the wafer in the chamber would be dusted and in an indeterminate state, and the rest of the wafers you’d have to sleuth out whether they were implanted or not. Sometimes you’d have a lot sitting in the end station and it wouldn’t be clear whether or not it had been run at all. At least in photolithography you could tell whether or not a wafer was patterned by looking at it.
Some mods in modded minecraft had that and it's a very punishing mechanic unless implemented well.
It eats all of your power and usually also very expensive items very quickly usually. Assume you have like 600RF/tick generated, common with certain generator constellations. 1 tick - 1024 RF and one input consumed, crafting fails due to not enough power. 1 tick wait, 1 tick, 1024 RF and one input consumed, ... This can void 10+ items / second, which can hurt very badly. Even for common items in fact.
It also tends to kick you while you're down, because it only kicks in if everything else is already failing. Then the only thing to continue functioning is the thing voiding your energy and your expensive items. Or even worse, if you did one miscalculation about your power grid, and then all of your resources are gone, often before you can react.
It can be interesting in the right packs, but it is Gregtech level hardness.
GT:NH has “easy mode” enabled in some regards - it won’t finish the craft but it WILL wait for power (actually keep trying) - so if you fix the power problems you can finish and not lose the mats.
GregTech doesn't use RF though, at least it didn't. Machines pull packets of amps through the wires from the generators/batteries, the whole system is pretty interesting. Also high-level circuits have to be manufactured in cleanrooms with a pretty complex tech chain.
Oh GTs power is absolutely not RF. Back in the day, even GTs power could be cruel though. You could over-volt your machines and thus void machines you spend literal days on crafting. And the cables in the process too. And you could lose your entire infrastructure once it rained and you had no roof :)
Out of curiosity, which mods are this cruel? I've been playing GT (modern) lately and even it doesn't void your machine's items unless you break the machine itself.
Oh this was in the days of yore of modded 1.4 and early 1.7. I don't remember specific mods, I just remember the pain and frustration of this happening.
I'm currently playing Stoneblock 4 and have been playing GT:NH and Nomifactory some time ago, and the more modern mods have learned a lot from those old janky things. Heck, back in the day every mod had a different power system and you needed a nonsense amount of conversion infrastructure, unless the modpack did a lot of work to combine all of this somehow, haha.
Power brownouts are pretty rare outside of the very early game. It's too easy and cheap to massively over produce power for that to really harm players outside the early game so I don't think there'd be much interest. Usually brownouts rapidly develop into full blown blackouts and black restarts as your miners reduce output during the brownout often leading to a reduction in incoming fuel leading to even less power being generated in a self consuming cycle.
Suppose you can start production with only 1 of each input required for a recipe, but to keep it going you need to keep feeding all of the inputs to finish it. If any of them run out, then the recipe fails, you lose the inputs, and the machine stalls.
This works better for high latency recipes (>10s) with lots of inputs, like low density structure, modules, and atomic bombs.
Mindustry has something similar with pumping various gasses/liquids through plumbing. If you accidentally mix them while building new lines, things stop working when your gases get mixed up forcing you to purge the line.
Someone needs to make the whole chip manufacturing process into a factorio like game and let the gamers optimize it, then build the factories around that.
seems like what is often downplayed or silent on American media is the cultural mismatch between TSMC taiwanese engineers and their american counterparts
so it always comes to those out of the loop as a bit of a surprise but from what I've read from individual Taiwanese workers and their feedback its clear that there is significant regret from one side.
and it doesn't seem to limited to just TSMC but another large company as of recent that receive icey reception for their large investment in America manufacturing.
i think this is a big reason why lot of these jobs simply wouldn't stay in america as the consumer would not be able to foot the costs added by "cultural premium" faster than what innovation can reduce.
TW workers have a majority of their compensation in bonuses, so the OT portion is quite small and many do not even bother to ask for it. The overall compensation between a TW and US engineer at TSMC is also significant. Not to mention the lowest paid hourly workers...where in TW they make 2-3X minimum wage, but in the US it's like 1.25X.
There are like half a dozen semiconductor manufacturers in Phoenix that were here before TSMC arrived. There's a robust pipeline from ASU to these same manufacturers. Can we please just stop with the nonsensical notion that "Americans don't know how to fabricate semiconductors"?
TSMC is a publicly traded company just like the others. I'm not familiar with their governance but Google tells me the largest owner (a state development fund) has 6%.
They have a special advantage because they don't compete with their customers, which leads to trust, which leads to customers paying for their R&D for them.
Intel on the other hand just kind of sucks at their job. Skill issue basically. (But they aren't /that/ far behind.)
its not that the USA can't produce semi-conductors. Its that semi-conductor production, at TSMC's scale (both in terms of number of units, yield rates, and depth) currently requires highly skilled workers to work a lot of their hours to "baby sit" the wafer production.
Maybe there is a world where TSMC can hire enough skilled workers and optimize processes enabling people to go home at 5p, but that is not currently the case.
Yes. This. So, yeah, essentially fundamentally incompatible with the US economy.
The US is going to have to heavily subsidize the payroll of thousands of very accomplished EEs/etc to make this work. By doing that they will also wreck the HW part of SV.
There isn't really a HW part of SV. Hardware engineers aren't paid well enough to live there in droves like programmers. There are some of course, but the ones I know are in San Diego or Bremerton or Israel.
Also, it's completely normal to run a factory 24/7. I think people are just impressed because TSMC is the only one they've read about?
(However, it's correct that a TSMC fab is the most advanced and complicated process on the planet.)
But first being reported now: It was only speculation on the financial reports before this. How quickly do they normally report disruptions like this?
I wouldn't think it would have to be too quickly since I've heard about fab disruptions from fires and such since the early 2000's. Probably just sometime after quarterly reporting to set the record straight? Why not in the report?
I also had the impression from the report that shareholders were miffed about this Q3 snag, so they had to publish this even though they were about to treat this as business as usual.
This was apparently a Linde installation custom built for TSMC in Arizona.[1] Nitrogen, oxygen, and argon are extracted from air on-site and purified. That's Linde's primary business; liquefying and distilling air. This isn't some little local company or a company operating outside their area of expertise.
Those gases are storeable, so it's surprising there wasn't enough tank capacity to deal with outages.
The site plan [2] shows "Gas Plant 1", and future "Gas Plant 2" and "Gas Plant 3". The gas plants are across a small road from the fab and feed the plant directly. Once Gas Plants 2 and 3 were built, there would be redundancy, but at this stage, there isn't a backup. The plan doesn't show a large tank farm, so they can't store gases in bulk.
[1] https://www.aztechcouncil.org/utility-company-makes-progress...
[2] https://semiwiki.com/forum/threads/tsmc-phoenix-arizona-fab-...
Unless things have changed a lot since I fled semiconductor manufacturing, you would still need silane tanks at least. I’m as surprised as you that they don’t have buffer tanks.
Linde is huge. They can produce and offer all gases in all available purity classes.
Is there some reason you'd want gasses in a lower purity class?
(Well, it's cheaper.)
Fire suppression, welding gases, etc.
Is this an ad?
I don't think Linde™ needs an add, everyone knows Linde™ is the most reliable partner in producing and storing gases in all available purity classes.
(joke off, it's probably not an add, but they were excited to share the reason you see Linde on all sorts of gas tanks all over the place. It's actually quite common and if you see it once you see it everywhere. )
Possibly but also plausible is that its a deep joke that everyone is in on.
When googling the company, the marketing slogan that comes up is "Linde is Everywhere" but that works on so many levels. They sell air, air is everywhere. Therefore Linde is everywhere.
They are a company that sells air: that stuff that people breathe. Forget this AI nonsense. Jensen has to constantly pull something out of rear to keep food on the table. These guys sell air. What a business. :)
This isn’t very big news. Issues occur during bring-up often. Linde’s processes are possibly so power intensive that failing over to generator power is not possible. TSMC is right to put Linde on notice since Linde should have a PFMEA and control plan to eliminate any root causes for downtime. I suspect in the long term TSMC has plans to insource this if the issue persists. Scrap happens sometimes during manufacturing, if the writer only has journalism experience and no manufacturing experience then they may not have a conceptual understanding of acceptable first pass yield. After all, the TSMC logo features failing parts!
In many ways I agree with you, but the problem statement (constrained/exhausted gas supply from vendor) makes it seems like this was not just line down, but the whole factory stopped for a few hours. Line down is a miserable migrane but still managable... while a whole factory stoppage makes a lobotomy seem like a good idea. It also sounds like there was not enough forewarning to park critical customer wafers in a "safe" stage of the process.
Even so, I also would still call this another monday at a semiconductor factory. Welcome! Here we play a nearly endless game of whack-a-mole. Here's your mallet and your towel. Now whack enough of the moles hard enough until they stop coming back (at least through the same holes). Beware the alpha moles.
By any road, I am surprised to see even this high-level perspective on a quality event disclosed to the mainstream public; I thought this was not standard practice. I enjoyed the read.
> forcing the facility to shut down for at least a few hours
> As a result, the company had to scrap thousands of wafers
Anything involving wet chemistry, photoresist, furnaces, etc. is very time-constrained. You can't let wafers sit around indefinitely. Certain process steps must be followed up very quickly to avoid scrap.
This is why you dont see redundant power for manufacturing lines. A 3nm line needs hundreds of megawatts to operate. You cant clear queued lots without a fully functional line. There's not much you could save by keeping part of the line operational.
TSMC has backup generators in their AZ fab. You actually have to have backup power or a few hundred millisecond blip could cause days or weeks of tool down time. You should see what happens when you lose the ability to keep a clean room at temp/humidity/airflow...it's weeks or months.
It didn’t happen, but the facilities team at the fab where I worked was seriously considering installing a flywheel to cover power bumps. What I don’t get about this story is how this actually happened. All our process gasses were out in a tank farm and we knew how much pressure we had. We would have stopped the line if there wasn’t enough to proceed. Were they separating air onsite or something?
I was very impressed by the modest little fab I worked at having thousands of lead acid batteries for momentary takeover, and 8 five-megawatt locomotive engines for longer term redundancy. Apparently their steady state usage was 25MW, which allowed still having a hot spare and concurrent downtime for two of the locomotive generator units.
Yes, Linde has an onsite plant and is building two more.
For some processes, stopping will botch the wafer. In the event of a gas shortage, do plants plan which lines to take down first, and which lines should complete a process step?
The way this worked at the fab where I was, was that facilities would have paged everybody, and whoever needed to hold wafers would do so. You could mark your equipment down or unavailable for a particular step. I don’t know what we would have done if it was “hey, we lost dry nitrogen a minute ago.” I think at that point you lose a lot of wafers in wet cleans.
In the case of a power interruption at the fab, consequences were highly dependent on the equipment and the unit process. A prolonged power interruption to diffusion was the worst case scenario. You’d have 150 wafers in the furnace, and any significant deviation from the nominal temperature profile meant they were all scrap. Worse, if the furnace cooled off, you had to scrap the quartz boat the wafers rode in, too. Other processes had a smaller blast radius but were even more of a headache to disposition. Implant, you’d lose beam and probably lose vacuum too. Then the wafer in the chamber would be dusted and in an indeterminate state, and the rest of the wafers you’d have to sleuth out whether they were implanted or not. Sometimes you’d have a lot sitting in the end station and it wouldn’t be clear whether or not it had been run at all. At least in photolithography you could tell whether or not a wafer was patterned by looking at it.
Good idea for a Factorio mod?
A new failure mode resets output progress back to zero if you lose power or some other input while crafting.
You could design circuit networks to cut power to non-essential systems so the rest of the factory can keep producing.
Some mods in modded minecraft had that and it's a very punishing mechanic unless implemented well.
It eats all of your power and usually also very expensive items very quickly usually. Assume you have like 600RF/tick generated, common with certain generator constellations. 1 tick - 1024 RF and one input consumed, crafting fails due to not enough power. 1 tick wait, 1 tick, 1024 RF and one input consumed, ... This can void 10+ items / second, which can hurt very badly. Even for common items in fact.
It also tends to kick you while you're down, because it only kicks in if everything else is already failing. Then the only thing to continue functioning is the thing voiding your energy and your expensive items. Or even worse, if you did one miscalculation about your power grid, and then all of your resources are gone, often before you can react.
It can be interesting in the right packs, but it is Gregtech level hardness.
GT:NH has “easy mode” enabled in some regards - it won’t finish the craft but it WILL wait for power (actually keep trying) - so if you fix the power problems you can finish and not lose the mats.
May or may not apply to multi blocks.
GregTech doesn't use RF though, at least it didn't. Machines pull packets of amps through the wires from the generators/batteries, the whole system is pretty interesting. Also high-level circuits have to be manufactured in cleanrooms with a pretty complex tech chain.
Oh GTs power is absolutely not RF. Back in the day, even GTs power could be cruel though. You could over-volt your machines and thus void machines you spend literal days on crafting. And the cables in the process too. And you could lose your entire infrastructure once it rained and you had no roof :)
I think GP just used RF as an example and was only referring to GT as a comparison in difficulty.
GT's system of only pulling power on-demand is very nice though; no wasting fuel
Out of curiosity, which mods are this cruel? I've been playing GT (modern) lately and even it doesn't void your machine's items unless you break the machine itself.
Oh this was in the days of yore of modded 1.4 and early 1.7. I don't remember specific mods, I just remember the pain and frustration of this happening.
I'm currently playing Stoneblock 4 and have been playing GT:NH and Nomifactory some time ago, and the more modern mods have learned a lot from those old janky things. Heck, back in the day every mod had a different power system and you needed a nonsense amount of conversion infrastructure, unless the modpack did a lot of work to combine all of this somehow, haha.
Power brownouts are pretty rare outside of the very early game. It's too easy and cheap to massively over produce power for that to really harm players outside the early game so I don't think there'd be much interest. Usually brownouts rapidly develop into full blown blackouts and black restarts as your miners reduce output during the brownout often leading to a reduction in incoming fuel leading to even less power being generated in a self consuming cycle.
I would apply it to inputs as well.
Suppose you can start production with only 1 of each input required for a recipe, but to keep it going you need to keep feeding all of the inputs to finish it. If any of them run out, then the recipe fails, you lose the inputs, and the machine stalls.
This works better for high latency recipes (>10s) with lots of inputs, like low density structure, modules, and atomic bombs.
Isn't that what spoilage is?
I thought of spoilage as a mechanic that punishes overproduction.
It's a constraint to process item within limited time (regardless of overproduction or power outage). Matching with the problem description.
Surely the reality might be much more complex (like... the yield/quality drop by time function?)
Mindustry has something similar with pumping various gasses/liquids through plumbing. If you accidentally mix them while building new lines, things stop working when your gases get mixed up forcing you to purge the line.
Someone needs to make the whole chip manufacturing process into a factorio like game and let the gamers optimize it, then build the factories around that.
Like Ender’s Game but instead of intergalactic shooting war it’s international chip war.
A video showing those steps, for the curious: https://www.youtube.com/watch?v=dX9CGRZwD-w
It's probably not 100% identical to TSMC's process.
seems like what is often downplayed or silent on American media is the cultural mismatch between TSMC taiwanese engineers and their american counterparts
so it always comes to those out of the loop as a bit of a surprise but from what I've read from individual Taiwanese workers and their feedback its clear that there is significant regret from one side.
and it doesn't seem to limited to just TSMC but another large company as of recent that receive icey reception for their large investment in America manufacturing.
i think this is a big reason why lot of these jobs simply wouldn't stay in america as the consumer would not be able to foot the costs added by "cultural premium" faster than what innovation can reduce.
Perhaps if the US workers earned OT like TW workers do, the "culture" gap would shrink.
TW workers have a majority of their compensation in bonuses, so the OT portion is quite small and many do not even bother to ask for it. The overall compensation between a TW and US engineer at TSMC is also significant. Not to mention the lowest paid hourly workers...where in TW they make 2-3X minimum wage, but in the US it's like 1.25X.
Is OT "overtime"? How is it legal not to pay overtime in any US factory? Unless they are salaried (exempt)?
yeah, overtime. My cousin is an engineer at TSMC (who worked both in Tainin and now in Arizona) and is w-2 exempt.
A lot of the workers there probably are exempt under American law.
I’m not an expert on Taiwanese labor laws but their list of exempt labor categories in the LSA is much shorter than the one in the American FLSA.
Hahaha. The work culture between TW and the US is night and day - and it isn’t flattering for the US.
How so? Are the Americans relatively lazy or just unwilling to put in tedious but necessary extended hours?
Spell it out. WTF.
There are like half a dozen semiconductor manufacturers in Phoenix that were here before TSMC arrived. There's a robust pipeline from ASU to these same manufacturers. Can we please just stop with the nonsensical notion that "Americans don't know how to fabricate semiconductors"?
American economics doesn't allow fabrication of semiconductors even if there is the know how.
Think about how Intel, who pioneered the know how, can't build cutting edge nodes in the levels that they need to make it profitable.
IBM had to sell their fabs to cater to the whims of "shareholders".
It's the greed of stockholders that you need to blame.
TSMC is a publicly traded company just like the others. I'm not familiar with their governance but Google tells me the largest owner (a state development fund) has 6%.
They have a special advantage because they don't compete with their customers, which leads to trust, which leads to customers paying for their R&D for them.
Intel on the other hand just kind of sucks at their job. Skill issue basically. (But they aren't /that/ far behind.)
its not that the USA can't produce semi-conductors. Its that semi-conductor production, at TSMC's scale (both in terms of number of units, yield rates, and depth) currently requires highly skilled workers to work a lot of their hours to "baby sit" the wafer production.
Maybe there is a world where TSMC can hire enough skilled workers and optimize processes enabling people to go home at 5p, but that is not currently the case.
Yes. This. So, yeah, essentially fundamentally incompatible with the US economy.
The US is going to have to heavily subsidize the payroll of thousands of very accomplished EEs/etc to make this work. By doing that they will also wreck the HW part of SV.
There isn't really a HW part of SV. Hardware engineers aren't paid well enough to live there in droves like programmers. There are some of course, but the ones I know are in San Diego or Bremerton or Israel.
Also, it's completely normal to run a factory 24/7. I think people are just impressed because TSMC is the only one they've read about?
(However, it's correct that a TSMC fab is the most advanced and complicated process on the planet.)
Some HN threads about the past incidents mentioned in OP,
https://news.ycombinator.com/item?id=17686310 ("Computer Virus Cripples Several Taiwan Semiconductor Plants (bloomberg.com)"—2018, 100 comments)
https://news.ycombinator.com/item?id=19214952 ("TSMC's Photoresist Material Incident: $550M Loss (anandtech.com)"—2019, 15 comments)
Does it further delay the launch of M5 Max/Ultra?
In September
But first being reported now: It was only speculation on the financial reports before this. How quickly do they normally report disruptions like this?
I wouldn't think it would have to be too quickly since I've heard about fab disruptions from fires and such since the early 2000's. Probably just sometime after quarterly reporting to set the record straight? Why not in the report?
I also had the impression from the report that shareholders were miffed about this Q3 snag, so they had to publish this even though they were about to treat this as business as usual.
the rebels have hit the tibonna gas supply I see
[dead]
guys is it me or sonnet 4.5 just became like 10x worst ?