From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS8075 40.64.0.0/10 X-Spam-Status: No, score=-3.1 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,RP_MATCHES_RCVD, SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=no autolearn_force=no version=3.4.0 Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-oln040092004062.outbound.protection.outlook.com [40.92.4.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 71EAA20281 for ; Tue, 23 May 2017 19:29:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=VVi3l6sAvpLs+MEFwATbY5Qy7vuiY2P9VGlcX/hHzYY=; b=BxSFNJU3Y3QQvqfhDOXA+5+DseHg9NlLz62TT6W/cRkjutYhkxMu00SVYqb3WtmiHnP2a6KEDsLDP9FK3EDpFMX6+9gs2KNYfyNq7qa/VSwqBQEYELEQZqwiH9329KfrZkv9EsugD5CAIpKjjPlcmQ7yW7lnmcg5u1iolLVJdHy9TVsZBwHuGHuhNX32QzG6JmH+2HYyjIDuHEFryy9UJyhez6dSlCe3mYQOQBGal7dpP1+T0uAgLaW929ALHiag+piB/b3IMq5ZkNESh93oHiM+x8Y+f+imVpfqbeaIzgx5+xKcoQV8FrOenCU4/1TM895TCx0/Sbv/iUHOj4xitA== Received: from BL2NAM02FT041.eop-nam02.prod.protection.outlook.com (10.152.76.59) by BL2NAM02HT132.eop-nam02.prod.protection.outlook.com (10.152.76.130) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.1075.5; Tue, 23 May 2017 19:29:13 +0000 Received: from BN6PR20MB1281.namprd20.prod.outlook.com (10.152.76.51) by BL2NAM02FT041.mail.protection.outlook.com (10.152.77.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1075.5 via Frontend Transport; Tue, 23 May 2017 19:29:08 +0000 Received: from BN6PR20MB1281.namprd20.prod.outlook.com ([10.173.159.151]) by BN6PR20MB1281.namprd20.prod.outlook.com ([10.173.159.151]) with mapi id 15.01.1101.019; Tue, 23 May 2017 19:29:08 +0000 From: Aakash Gupta To: "unicorn-public@bogomips.org" CC: "unicorn-public@bogomips.org" Subject: Re: Master Process Reaping Worker with No message Thread-Topic: Master Process Reaping Worker with No message Thread-Index: AQHS0/q+6nwmyVJoCUOoZEnn7wwx4w== Date: Tue, 23 May 2017 19:29:08 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: bogomips.org; dkim=none (message not signed) header.d=none;bogomips.org; dmarc=none action=none header.from=outlook.com; x-incomingtopheadermarker: OriginalChecksum:9A68CF8C2625116276FD845A98B8D2DAE6D101D2CCCCB650D38F7FBA83F6F3EB;UpperCasedChecksum:16767F8309080DC0795EF706282F1207A3C498F5A14C328D705F9AC3ABE99DA5;SizeAsReceived:8136;Count:43 x-tmn: [AWKOZVKJNO0BhsWoE0lGHal69BVsTFmEzg2cD0OUriRc1522USh9rg==] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;BL2NAM02HT132;5:kxxsa4xkfAgvzkXlL2rGjO7Y75nQ/Lqi19cOT7gWxB4pfJ7B763t4SVH7rhaIyJD20PKYUSKtY7ojPDHO5IceF+mRRfu0SJNxukpnMN7UKPo4nWKPvqx56skdhcmvqOT/ILv82y9jr9MVNm7HEMLw3g3O2v4AMjacdUDFy0UsZY=;24:0Acn/jAtzmiZrZ6f7rUYLBBAhvU7/DoKHALiUa2wQx/1E0/A9kv8QxSBjSfwHR/tOGR6spH+jbZx3Jb7tiESNv/xlUrl7sjK5K0bE4YJtsQ=;7:TprUCIn6kf63c4mH/Beir94wroqj7jUNO4/NAn7Fs26aa87mDVZmSKLpd/1kN3OOkl3lJ/LxQAzztS/IqkHCR10jUXMuBaAOfCR30jJqYdO/KM9T2DIekJWmquXZlExTFY2eKD7JlioJBbhUYBFqn54wR/zfqKAed7szh4uO165WtN0r7KauyjQoZrk/6PHqFsfNsIJ2ZnpPjxfqdsuLFUV95A8/8t4hsWlj4JrjueU6P0asQG1OJj0EpF6usoS8NedzeGzU+UIqHbq4LghuwvMj7kmIuUC3miUidQLLX9eNwtDSVcZFoFQY4drf2FVf x-incomingheadercount: 43 x-eopattributedmessage: 0 x-forefront-antispam-report: EFV:NLI;SFV:NSPM;SFS:(7070007)(98901004);DIR:OUT;SFP:1901;SCL:1;SRVR:BL2NAM02HT132;H:BN6PR20MB1281.namprd20.prod.outlook.com;FPR:;SPF:None;LANG:en; x-ms-office365-filtering-correlation-id: 4d2a3a7e-8927-448b-78ce-08d4a211fbce x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(201702061074)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322274)(1601125374)(1603101448)(1701031045);SRVR:BL2NAM02HT132; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(444000031);SRVR:BL2NAM02HT132;BCL:0;PCL:0;RULEID:;SRVR:BL2NAM02HT132; x-forefront-prvs: 0316567485 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-originalarrivaltime: 23 May 2017 19:29:08.7968 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2NAM02HT132 List-Id: Thank a lot Eric.=A0 I am using Ruby 2.3.3 and Unicorn 5.3.=A0 "It sounds like somebody else is sending SIGKILL to a worker..." This line = of yours saved my server.=A0 Actually when I uploads a file, it first goes to /tmp directory and then on= e of the worker processes of unicorn uploads it to S3 from there. So, earli= er when I was trying to fix timeout connection error of unicorn, I increase= d it to 20 minutes and the problem was solved temporarily.=A0 But the problem was that the files that were uploaded temporarily to /tmp d= irectory, those files do not get deleted from /tmp after they are uploaded = to S3. So, I wrote a script using tmpreaper that cleans the /tmp directory = recursively and removes the files that are 30 min older and added that to c= ron service.=A0 After this, as a result of this script, all the files that were required to= be removed, were removed from /tmp directory but again a problem occurred.= When I used "df -h" command to check whether the files are actually delete= d or not, it showed that the space that was taken by uploaded files on my d= isk has not been freed :( . So, I searched internet and got to know that an= y other process was still using those files, that was why the space on the = disk was not freed.=A0 I used ' lsof +L1 | grep "deleted" ' command and found that a process named= 'bundle' is still using these deleted files. So, I again wrote a script th= at was killing the process that was using a deleted file.=A0 Script was usi= ng this command " lsof +L1 | grep 'deleted' | awk '{print $2}' | xargs kill -9 " Today, after reading your words, I finally realised that my script was kill= ing the unicorn worker itself because the command that I was using was kill= ing the parent of "bundle" process. As "bundle" was a child of Unicorn work= er, unicorn worker was being killed. So, thanks a lot Eric :)=A0 I was finally able to solve this error because of the direction and hint th= at you gave me. But the only thing that I need to find is that what is caus= ing uploaded files to persist in /tmp directory.=A0=A0=A0