1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:

11:44:57 	OdiliTime 	what's the m.8ch.net config like?
[...]
11:48:38 	JB 	these are the average VIRT / RES / SHR memory stats for the php children running on poal.me: 447m 5056 1916
11:48:52 	JB 	i have 300 children running
11:48:58 	JB 	8 GB ram on the box
11:49:25 	JB 	only 2.7 GB of the 8 GB used according to free
11:52:58 	JB 	but for this, if you are still messing with it, OdiliTime says that with nginx you should not use the overriding fastcgi_params i told you about, so i guess you should remove those two lines and just edit the pm.max_children value in php-fpm.conf like normal instead
11:54:19 	OdiliTime 	n-tech: did you install via compile, ports or packages?
11:54:32 	@n-tech 	I didn't install anything, he did
11:54:38 	@n-tech 	I think he made them manually.
11:54:43 	OdiliTime 	copypaste did?
11:54:56 	JB 	if you see no way to make extra children happen, then it's time to look into ways to optimize the db queries i think, so the few children there are can handle requests as quickly as they come in, otherwise you will keep seeing a handful of children sitting idle as they wait for db queries to complete while nginx has a growing number of http connections on hold waiting to be handed off to the busy php-fpm children
11:55:27 	OdiliTime 	queries are fine, 30ms
11:55:55 	JB 	ahh knocked down the 80ms-900ms then?
11:56:45 	OdiliTime 	no
11:57:06 	OdiliTime 	he's got slow query logs, showing good query execution times
11:57:52 	CQO 	postgres's query profiling includes time spent in lockwait, right?
11:58:21 	OdiliTime 	it must include that, what DB wouldn't
11:59:11 	CQO 	some can get a little fucky on their defaults, but I assume locking's already been checked as a possible cause
12:00:31 	CQO 	given the servers not breaking a sweat but still running like chilled molasses, locking or other mutexy stuff would be where I'd look first
12:00:56 	OdiliTime 	yea even the static files are slow
12:00:56 	Drybones 	give the server a red bull
12:00:59 	OdiliTime 	something is weird
12:02:17 	CQO 	no idea yet what part of the stack the delay is coming from, then?
12:02:30 	OdiliTime 	there's 2 separate servers taking 40s to serve shit
12:02:36 	JB 	isn't db stuff largely relaxed due to redis though
12:02:43 	OdiliTime 	that too
12:02:52 	JB 	what's the uplink speed on this box
12:03:28 	OdiliTime 	guessing 1gbps if it's like 8archive
12:03:35 	OdiliTime 	full duplex
12:04:18 	CQO 	wouldn't be link, not with the nature of the degradation being seen
12:04:43 	OdiliTime 	yea I think we'd see more packet loss but cloudflare's in the way
12:04:43 	JB 	i'm guessing only warmtires can check the consumption graph provided by the host
12:05:01 	OdiliTime 	8archive is on the same uplink and it seems fine, no packet loss
12:05:31 	CQO 	that static files are seeing the same problems is pretty suss
12:05:45 	OdiliTime 	warmtires?
12:06:03 	JB 	hotwheels
12:06:08 	OdiliTime 	oh kek
12:06:14 	CQO 	he of the burning chariot
12:06:48 	JB 	if cloudflare is not set to cache anything for m.8ch.net then maybe that is enough of a problem for the uplink to be completely saturated
12:07:07 	OdiliTime 	it's not saturated though
12:07:14 	JB 	if that is the cause, then it is happening at the same time as the php and/or db thread locking problem
12:07:21 	JB 	oh
12:07:24 	OdiliTime 	8ch.net is fine on the same link
12:07:44 	CQO 	how much info do we actually have?
12:07:51 	CQO 	and how do we get more?
12:08:00 	JB 	did we get a report from n-tech on whether tail -n 100 /var/log/messages said anything strange
12:08:03 	OdiliTime 	n-tech posted some config files
12:08:14 	OdiliTime 	and he's been answering specific questions
12:08:20 	@n-tech 	https://ghostbin.com/paste/vh627
12:08:21 	OdiliTime 	JB: not really
12:08:26 	JB 	i have a few other ideas for correcting network problems on my cheat sheet from bullshit i had to deal with when scaling a site to handle 400 requests per second at peak hours
12:08:28 	@n-tech 	This is my total writeup
12:08:32 	@n-tech 	I'm placing the datacenter order now
12:08:37 	@n-tech 	We may go with Ubuntu
12:08:38 	@n-tech 	Discuss
12:08:54 	@n-tech 	copy is reluctant to use debian 8 because the dev na heroed
12:08:56 	@n-tech 	an heroed*
12:09:04 	@linear 	ubuntu or debian should both go fine
12:09:09 	OdiliTime 	debian or ubunutu are fine by me
12:09:12 	@linear 	also he was killed tbh