SteamCMD can't retrieve content

Hello,

I am having an issue with SteamCMD and I'm hoping someone could give me some hints on appropriate places to look for a resolution for this problem.

I was running a private dedicated server for a Steam game called Insurgency. This was working perfectly under my original iteration of my server. It was FreeBSD 10.4 with the linux_base-c6. Unfortunately at the time I was a total noob to FreeBSD and made some very poor partitioning choices during the initial configuration. Recently I reached an impassible limit on the server and opted to wipe and reinstall with all this hindsight I have accrued.

The system is now on FreeBSD 11.1 and Linux_base-c7.

SteamCMD will run and I can use various commands within SteamCMD which all seem to work as intended (ie Logging in, forcing install directory, starting an installation). The problem arises when I instruct SteamCMD to install the Insurgency dedicated server. After entering the command to install I am met with the following error:

Code:
 Steam>app_update 237410 validate Update state (0x0) : Timed out waiting for update to start, bailing. Error! App '237410' state is 0x2 after update job.

SteamCMD generates some logs and there is some telling info there:

Content.log: You can see here it complaining about no connection to content servers
Code:
[2018-05-20 12:20:02] Loaded 0 apps from install folder "/home/pking/Steam/steamapps\appmanifest_*.acf".
[2018-05-20 12:21:48] Failed getting content server list from directory (status: 0)
[2018-05-20 12:22:15] Loaded 1 apps from install folder "/usr/home/pking/INS/steamapps\appmanifest_*.acf".
[2018-05-20 12:22:23] AppID 237410 scheduler update : Priority First, legacy=no, restore="", timeSinceLastPlayed=1526833343, appDisableSecondsRemaining=0
[2018-05-20 12:22:23] AppID 237410 state changed : Update Required,Update Queued,
[2018-05-20 12:22:23] AppID 237410 state changed : Update Required,Update Queued,Update Running,
[2018-05-20 12:22:23] AppID 237410 update changed : Running,
[2018-05-20 12:22:23] AppID 237410 update changed : Running,Reconfiguring,
[2018-05-20 12:22:23] Empty CS address list - cannot download anything!
[2018-05-20 12:22:23] AppID 237410 update canceled : No servers found for depot 1006, manifest 440706658004308074 (No connection to content servers)
[2018-05-20 12:22:23] AppID 237410 update changed : Running,Reconfiguring,Stopping,
[2018-05-20 12:22:23] AppID 237410 update changed : Running,Stopping,
[2018-05-20 12:22:23] AppID 237410 update changed : None
[2018-05-20 12:22:23] AppID 237410 state changed : Update Required,Update Queued, (No connection to content servers)
[2018-05-20 12:22:23] AppID 237410 state changed : Update Required, (Update delayed for 300 secs)
[2018-05-20 12:22:23] AppID 237410 scheduler finished : removed from schedule

Connection.log: Again you can see multiple attempts to contact Steam CM servers which i believe are content servers that provide game files. In all cases it seems like the connection is timing out.
Code:
[2018-05-20 10:37:10] Log session started
[2018-05-20 10:37:10] [0,0] SetSteamID( [U:1:0] )
[2018-05-20 10:46:51] [0,0] SetSteamID( [a:1:0] )
[2018-05-20 10:46:51] [0,0] SetSteamID( [a:1:0] )
[2018-05-20 10:46:51] [0,0] Server says 0% of connections should be websockets, we rolled 45 - using UDP as default.
[2018-05-20 10:46:51] CCMInterface::YieldingConnect -- no CMs cached locally, calling ISteamDirectory/GetCMList web api. . .
[2018-05-20 10:46:51] CCMInterface::YieldingConnect -- web API call failed (status = 0)
[2018-05-20 10:46:51] [0,0] PingWebSocketCM() (CM04-IAD.cm.steampowered.com:443) starting...
[2018-05-20 10:46:51] [0,0] PingWebSocketCM() (cm02-lax.cm.steampowered.com:27021) starting...
[2018-05-20 10:46:51] [0,0] PingWebSocketCM() (cm02-lax.cm.steampowered.com:443) starting...
[2018-05-20 10:46:51] [0,0] PingWebSocketCM() (cm01-lax.cm.steampowered.com:27021) starting...
[2018-05-20 10:46:51] [0,0] PingWebSocketCM() (CM04-IAD.cm.steampowered.com:443) failed talking to cm (timeout/neterror - Invalid)

No firewall is configured on this machine and the system is connected directly to my ISP modem. If any packets are being denied then its being done by my ISP which I believe is unlikely because as I mentioned, this worked fine a few weeks ago prior to my wipe and re-install.

I believe the issue is networking related but from the position of permission to access the resources vs something actively denying traffic.

When I initially launch SteamCMD I am met with the following error:
Code:
Loading Steam API...opensslconnection.cpp (1490) : Assertion Failed: unable to load trusted SSL root certificates

If SteamCMD is unable to load the trusted certificate on my system I imagine that might prevent connection to their servers to retrieve game files.

I have tried removing and reinstalling CA-root-nss but that didn't seem to have any effect. Some reading earlier suggested to me that steam only looks for certificates in /etc/ssl/certs and perhaps my certificate store may be located in a different directory. I think symbolic links exist in part to resolve an issue like that. I've not used them yet so I will have to get on my system and test.

Anybody have any other tips or suggestions, tools i can leverage to better understand why the networking side of this is failing?

Any suggestions at all are welcome.

Thanks!
 
Code:
Steam>app_update 237410 validate Update state (0x0) : Timed out waiting for update to start, bailing. Error! App '237410' state is 0x2 after update job.
I get those on my Linux game server too. It means the Steam client lost connection to the Steam servers. That SteamCMD application is quite finicky and regularly craps out during updates, especially big ones.
Anybody have any other tips or suggestions, tools i can leverage to better understand why the networking side of this is failing?
I get the exact same issues on a Linux server, so this problem isn't specific to FreeBSD. All I can say is, try, and try again.
 
When you say "try, and try again", are you just continuously trying without delay or are you trying once and then waiting 15min, 2 hours?

Appreciate the feedback. I was very close to installing a linux distro so I could see the experience on SteamCMD's native platform. What you say doesn't surprise me in the least. Actually my first experience with SteamCMD was similar as far as unreliability goes. This time however, I'm certain there is something preventing successful communication to Steam's servers. I've been troubleshooting this pretty heavily (because I really want my Insurgency server back lol) and have tried launching this installer probably 40-50 times under different configurations, with the same result.

I've got quite a few ideas to test out this weekend. I'll let you know how I make out.
 
I have tried removing and reinstalling CA-root-nss but that didn't seem to have any effect. Some reading earlier suggested to me that steam only looks for certificates in /etc/ssl/certs and perhaps my certificate store may be located in a different directory. I think symbolic links exist in part to resolve an issue like that. I've not used them yet so I will have to get on my system and test.

Try sudo ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem.
 
The link was actually already present. Tried many other things on the weekend with still no luck. I wiped the system and went back to freebsd 10.4 and linux_base-c6 (same configuration I had when this last worked). Still the same issue persists.

So likely Steam changed something recently that has broken this for me. I'm going to quickly install Debian and deploy steamcmd on its native platform to see if it works without issue. I'm betting it will probably work fine. I believe the issue under FreeBSD is either permission related (probably execution) or some sort of dependency or other prerequisite not met.

If it doesn't work under Debian then I can probably assume the issue is ISP related but I'd say that is unlikely.

Steam did confirm that based on my error logs the issue is that I am unable to connect to their content servers.

It's possible in my original working setup I did something that resolved this issue. More likely I believe steamcmd was changed to include some sort of extra security or efficiency.

I'll keep this thread posted with my results. Determined to sort this out.
 
If it's a dedicated server (not a VPS?) it may be an idea to test things in a VM using bhyve(8) for example. That will allow you to quickly switch back and forth between the different operating systems. It may prove useful, at least for trying to figure things out.
 
failed talking to cm (timeout/neterror - Invalid)

Ok, here is a workaround:

1. steam_epoll_fix.c:
C:
#define _GNU_SOURCE

#include <dlfcn.h>
#include <errno.h>
#include <stdio.h>
#include <sys/epoll.h>

static int (*libc_epoll_ctl)(int, int, int, struct epoll_event*) = NULL;

int epoll_ctl(int epfd, int op, int fd, struct epoll_event* event) {

  if (!libc_epoll_ctl) {
    libc_epoll_ctl = dlsym(RTLD_NEXT, "epoll_ctl");
  }

  int err = libc_epoll_ctl(epfd, op, fd, event);
  if (err == -1) {

    if (op == EPOLL_CTL_MOD && errno == ENOENT) {
      errno = 0;
      return epoll_ctl(epfd, EPOLL_CTL_ADD, fd, event);
    }

    perror(__func__);
  }

  return err;
}

2. /compat/linux/bin/cc -Wall -m32 -std=c99 -shared -fPIC -I/compat/linux/usr/include -L/compat/linux/lib -o steam_epoll_fix.so steam_epoll_fix.c (gcc can be installed from linux-c7-devtools or linux-c6-devtools)

3. env LD_PRELOAD=steam_epoll_fix.so steamcmd
 
Back
Top