FIXED: Rollback issues on all Rust servers
We found out that some of our servers (and worldwide most of them) were having issues. After a restart, the server rolled back a couple of hours. Server owners blamed their hardware, but at the end a bug was found in the Rust servers causing it not to save, although it said it did.
Server owners with modded servers were able to fix this with a hotfix, but non-modded servers (vanilla) were bound to rollback again because they could not save the worldfile containing all information like buildings, entities and stuff.
Official statement from Rust-dad Helk:
Due to a bug the server save function has stopped working resulting in some servers not saving for several hours now.
To apply a fix we must restart servers which cannot be saved. Clearly this will result in a rollback to the last successful server save.
We’re working on a last ditch solution to prevent it but we’re looking at a rollback to yesterday on most official servers. We’ll know more in about 20 minutes.
Blueprints will not rollback.
We have updated all our servers and besides a really short rollback, we are up and running again with a secured server.
Please keep in mind: Rust is an Alpha game. The devvers are really working hard on fixing issues that may pop up. Bugs may be introduced during new updates, but the periodic updates are really a good thing. As soon as they find out an issue, they are the first to fix it. Respect to them.
Happy Rusting and apologies for any inconvenience.
UPDATE
The bug was introduced after code was written for the workbench experimentation. It prevented the server to save when a save was initiated while anyone was using the workbench. Unfortunately this happened with some of our servers as well yesterday.
Due to the recent issues (there were two now in one week), Facepunch is evaluating their weekly update processes. See below.
Statement Facepunch:
Long story short: I wrote some code for the workbenches that ended up preventing the server from saving the game if it initated a save while someone was in the process of experimenting.
We tried long and hard all night to come up with a solution, including writing injector programs to modify the memory of the server to bring its save functionality back to life, but to no avail.
Any server that did not have an unlucky timing should be fine, but servers that saved during that 5 second window are screwed.
When stuff like this happens, it makes me second guess our ‘weekly patch’ approach. While we have a pretty good track record, moving forward and out of early access this kind of thing isn’t really acceptable. Even when we don’t have a major wipe-requiring issue, sometimes we have to cram stuff into the patch that probably isn’t ready for primetime so you guys are getting sub-standard content. I think we really need to re-evaluate our decisions here and consider putting a feature-locked patch on a separate branch that can be tested for a solid week before being pushed out.
We’re so sorry.