From: "Mulvaney, Mike" <MMulvaney@bna.com>
To: "unicorn-public@bogomips.org" <unicorn-public@bogomips.org>
Subject: TeeInput leaks file handles/space
Date: Wed, 22 Apr 2015 16:42:51 +0000 [thread overview]
Message-ID: <CY1PR0301MB078011EB5A22B733EB222A45A4EE0@CY1PR0301MB0780.namprd03.prod.outlook.com> (raw)
When I upload large files to my unicorn process, TeeInput is streaming them through a @tmp instance variable which writes to /tmp/0.123456789 (some random number). The file is immediately deleted but the file handle is not closed, so the files are not really deleted by the file system.
The files are eventually deleted when GC runs, but before GC runs they continue to take up space. You can see this in lsof output, for example:
ruby 2783 webuser 23u REG 0,17 6128086 3556917 /tmp/0.04249158625633187 (deleted)
This can cause problems if you have big files and a small /tmp, such as a tmpfs disk mounted in ram. If someone sends in several 100MB files, you could easily get 2-3 open files for each of 6 unicorn processes, which would take up 1200MB of disk space until GC decides to run.
I looked into fixing this but it doesn't look easy. I can reach into the TeeInput variable and close out the @tmp instance variable in my application, and that does fix the problem. But obviously that is not a good solution. I think there would have to be some kind of "close" method on http_request that would close out all the open resources such as these files.
-Mike
next reply other threads:[~2015-04-22 16:42 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-04-22 16:42 Mulvaney, Mike [this message]
2015-04-22 18:38 ` TeeInput leaks file handles/space Eric Wong
2015-04-22 19:10 ` Mulvaney, Mike
2015-04-22 19:16 ` Eric Wong
2015-04-22 19:24 ` Mulvaney, Mike
2015-04-24 3:08 ` Eric Wong
2015-04-24 11:56 ` Mulvaney, Mike
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/unicorn/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CY1PR0301MB078011EB5A22B733EB222A45A4EE0@CY1PR0301MB0780.namprd03.prod.outlook.com \
--to=mmulvaney@bna.com \
--cc=unicorn-public@bogomips.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).