Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected SIGTERM handling #2653

Closed
qwelias opened this issue Jun 24, 2020 · 19 comments
Closed

Unexpected SIGTERM handling #2653

qwelias opened this issue Jun 24, 2020 · 19 comments
Assignees

Comments

@qwelias
Copy link

qwelias commented Jun 24, 2020

pnpm version: 5.2.6

Code to reproduce the issue:

// sigterm.js
process.on('SIGTERM', (...args) => {
    console.log(...args);
    setTimeout(process.exit, 5000, 123);
});
setInterval(console.log, 1000, 1);
// package.json
...
"script": {
  "sigterm": "node sigterm.js"
}
...
  1. pnpm run sigterm
  2. in another tty kill -TERM <pid_of_pnpm_run_sigterm>

Expected behavior:

pnpm run sigterm upon receiving SIGTERM should pass the signal to child and wait for it to exit, pass 123 exit code to parent.

Actual behavior:

pnpm run sigterm upon receiving SIGTERM passes signal to child, but does not wait for it to exit, hence child's exit code is lost (and child is potentially running detached?).

Additional information:

EDIT:

  • change 666 to 123 because it's capped at 154
@zkochan
Copy link
Member

zkochan commented Jun 25, 2020

I am not sure why it works differently with pnpm as we use a fork of npm's library for running the scripts. And it has the following code:

process.once('SIGINT', procInterupt)

  function procInterupt () {
    proc.kill('SIGINT')
    proc.on('exit', () => {
      process.exit()
    })
    process.once('SIGINT', procKill)
  }

https://github.com/zkochan/lifecycle/blob/latest/index.js

@qwelias
Copy link
Author

qwelias commented Jun 25, 2020

@zkochan
it seems like npm just kills itself upon SIGTERM according to relative source file, but I'm not sure if that's the whole picture.

Here's an example output comparison of npm vs pnpm:
(The logs also include terminal prompt which shows last non-zero exit code after timestamp.)

`npm run sigterm` gets SIGTERM
qwelias@schibmaze ~/prj/stuff  (1.0.0)                                                                                                                                    ⬡ 12.18.1 [10:48:20]
$ npm run sigterm  

> [email protected] sigterm /home/qwelias/prj/stuff
> node sigterm

1
1
1
1
1
SIGTERM 15
1
1
1
1
1
npm ERR! code ELIFECYCLE
npm ERR! errno 123
npm ERR! [email protected] sigterm: `node sigterm`
npm ERR! Exit status 123
npm ERR! 
npm ERR! Failed at the [email protected] sigterm script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/qwelias/.npm/_logs/2020-06-25T08_48_37_781Z-debug.log

qwelias@schibmaze ~/prj/stuff  (1.0.0)                                                                                                                                ⬡ 12.18.1 [10:48:37] 123

npm exits only after it's child is exited, exit code passed.

`pnpm run sigterm` gets SIGTERM
qwelias@schibmaze ~/prj/stuff  (1.0.0)                                                                                                                                ⬡ 12.18.1 [10:55:27]
$ pnpm run sigterm  

> [email protected] sigterm /home/qwelias/prj/stuff
> node sigterm

1
1
1
1
1
1
SIGTERM 15

qwelias@schibmaze ~/prj/stuff  (1.0.0)                                                                                                                                    ⬡ 12.18.1 [10:55:46]
$ 1
1
1
1
1


qwelias@schibmaze ~/prj/stuff  (1.0.0)                                                                                                                                    ⬡ 12.18.1 [10:55:55]

pnpm exited immediately after forwarding SIGTERM, while the script continued running orphaned and attached to stdio, exit code was lost

Copy link

Dec 15, 2021

A general demonstration of the issue with SIGINT:

image

Copy link

Hi, was just wondering if this was being looked at. I've reverted to using npm run on our project and pnpm for everything else. pnpm is perfectly fine for us in every other sense and I see new features being released frequently that seem like they tackle edge cases that I've never run into in 1.5 years of using this library. In my humble opinion, this anomalous handling of system signals should be of higher priority.

@zkochan
Copy link
Member

zkochan commented Mar 18, 2022

I did add it to a priority list but I have too much to do. And it is pretty hard for me to concentrate now, since I am in Ukraine.

I am OK to accept a contribution that makes it work the same way as it works in npm.

Copy link

Hope you are safe.

@DylanCulfogienis
Copy link

I'd be happy to contribute a PR for this if someone can just point me vaguely in the direction of where this is handled in pnpm. Already see the link above for npm.

Our team's been slowed down by this one for about a month and we'll have some extra time in the coming month or so to address lower-priority stuff for our project, including resolving this issue. Happy to give back a little to a project that's been instrumental to the success of our own.

@zkochan
Copy link
Member

zkochan commented Oct 19, 2022

The issue is probably in this package: https://github.com/pnpm/npm-lifecycle

@DylanCulfogienis
Copy link

Sweet, I'll take a look soon here.

@mark-omarov
Copy link
Contributor

Hey @DylanCulfogienis , are you still working on it? If there's any progress - please share.

@DylanCulfogienis
Copy link

@mark-omarov Afraid I haven't gotten to it yet- This work ultimately has to fit around the high-priority work for this project for me to justify it to our client.

If someone else stumbles across this and wants to give it a try, you're welcome to!

That being said, I will likely have time to take a look at this in late November/December. We're starting to clear out the high priority stuff, and this will be one of the first things I work on after that's said and done.

@mark-omarov
Copy link
Contributor

mark-omarov commented Nov 18, 2022

@DylanCulfogienis thanks for the update. I have tried to figure it out, no luck yet.

All I can tell so far is that on termination I see three events fired - process.once('SIGTERM', procInterupt), proc.on('close', ...), and process.on('exit', procKill) (file). For some reason, the process doesn't wait for its child indeed.
I've tried to ignore all except SIGINT, but the result was the same.

It's getting late, but I will take another look sometime later, and give an update if I figure something out.

P.S.
Also, the same behavior can be found with pnpm exec, but that works differently, there's no use of the mentioned lifecycle, I will try to find some time to check that out as well.

Update 24/11/2022:
I gave up for the time being. 😢 Might try again after end of the year holidays, no promises though.

@paescuj
Copy link
Contributor

paescuj commented Dec 11, 2022

It seems like the issue described in the OP has been solved in the meantime (at least on macOS, can others confirm as well?):

pnpm run sigterm

> monorepo-root@ sigterm <redacted path>
> node sigterm.js

1
1
1
SIGTERM 15
1
1
1
1
1

 ELIFECYCLE  Command failed with exit code 123.

(compare with output in #2653 (comment))

@zkochan
Copy link
Member

zkochan commented Dec 11, 2022

Yes, it was fixed in v7.18.0

@zkochan zkochan closed this as completed Dec 11, 2022
@ivstiv
Copy link

ivstiv commented Feb 8, 2023

I am experiencing the broken behaviour that is said to be fixed. Just tried the OP's code to reproduce it and I ended up with a zombie process no matter whether I used pnpm, yarn or npm. Can anybody else confirm this or am I doing something stupid?

Versions used:
[email protected]
[email protected]

@qwelias
Copy link
Author

qwelias commented Feb 9, 2023

@ivstiv
[email protected]
[email protected]

works fine, but keep in mind this may also depend on your OS

Copy link

// package.json
...
"script": {
  "sigterm": "node sigterm.js"
}
...
// sigterm.js
process.once('SIGTERM', () => {
  console.log('SIGTERM');
  
  setTimeout(() => {
    process.exit(0);
  }, 1_000);
});

process.once('SIGINT', () => {
  console.log('SIGINT');

  setTimeout(() => {
    process.exit(0);
  }, 1_000);
});

setInterval(() => {
  console.log('>>>');
}, 1_000);
$ node sigterm.js
>>>
^CSIGINT
>>>

Expected ^

$ pnpm run sigterm

^C ELIFECYCLE  Command failed.

Not expected ^

@zkochan appears to remain an issue

@paymog
Copy link

paymog commented Nov 7, 2023

I'm still seeing this issue on pnpm 8.10.0

@benweint
Copy link

benweint commented Dec 4, 2023

Since this issue is marked as closed, I filed #7374 as a new issue describing the problem that @gajus described above.

ardasevinc added a commit to ardasevinc/fastify-graceful-shutdown that referenced this issue Jul 3, 2024
extra: this fixes [pnpm issue 2653[(pnpm/pnpm#2653)
and see hemerajs#3
as the root cause of why there is this fork.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

10 participants