A server-side update went out today, patch notes -
Connectivity Post-Mortem:
tl;dr: Our team worked throughout the Easter weekend and around the clock to resolve the server issues players were experiencing. We completely understand how frustrating this experience will have been especially given the huge amount of players eagerly anticipating the launch. We had enough server scaling capacity but our externally hosted database was seeing issues that only appeared at extreme loads.
We’re committed to full transparency with you. Today, just as we have been over the past year.
So we won’t give you the expected “server demand was too much for us”.
We were in fact debugging a complex issue with why some metric calls were bringing down our externally hosted database. We did not face this issue during the demo launch earlier this year.
Our database is used to hold onto everyone’s gear, legendaries, profile and progression.
Tech-heavy insight:
We managed to understand that many server calls were not being managed by RAM but were using an alternative data management method ("swap disk"), which is too slow for the flow of this amount of data. Once this data queued back too far, the service failed. Understanding why it was not using RAM was our key challenge and we worked with staff across multiple partners to troubleshoot this.
We spent over two days and nights applying numerous changes and improvement attempts: we both doubled the database servers and vertically scaled them by approximately 50% (“scale-up and scale out”). We re-balanced user profiles and inventories to new servers. Subsequent to the scale-up and scale-out, we also increased disk IOPS on all servers by approximately 40%. We also increased the headroom on the database, multiplied the number of shards (not the Anomalous kind) and continued to do all we were able to in order to force data into RAM.
Each of these steps helped us improve the resilience of the database when under extreme loads, but none of them were the "fix" we were looking for.
At this moment in time we are still waiting for a final Root Cause Analysis (RCA) from our partners, but ultimately what really helped resolve the overloading issue was configuring our database cache cleaning, which was being run every 60 seconds. At this frequency the database cache cleaning operation demanded too many resources which in turn led to the above mentioned RAM issues and a snowball effect that resulted in the connectivity issues seen.
We reconfigured the database cache cleanup operations to run more often with fewer resources, which in turn had the desired result of everything generally running at a very comfortable capacity.
All of this has enabled the servers to recover and sustain significantly more concurrent user loads.
Multiplayer Stability and Connection:
Co-op and multiplayer sessions are a separate matter from solo play connectivity.
Launching a multiplayer session, requires multiple different partners and services. For example, matchmaking and queuing are hosted by one service partner, while the peer-to-peer relay is handled by another. Other components are hosted by other partners. This is a normal setup for multiplayer games but is important to mention because when we attempt to track multiplayer or co-op issues, knowing exactly what issue appears at what point is vital.
The upcoming patches should help alleviate crashes in multiplayer as well as improve stability and speed. Because of the complexity of the multiple systems involved, we will continue to improve and iterate upon these systems in order to deliver you a better multiplayer experience. In most cases, this can and has already been done via the backend without patching the game, but it is something we are keeping a close eye on.
Re-balancing:
This re-balancing work can be implemented through backend updates, does not require a patch, and will be run during today’s maintenance window.
IMPORTANT: Balancing is done by a separate team to those who are working hard on preparing the patches. Re-balancing does not impact our progress towards patches or reduce our ability to resolve bugs.
General Balance Notes:
Historian, bounty hunt, and monster hunt quests no longer grant Legendary items for each subsequent completion
While rewarding players for completing all the Historian, bounty hunt, and monster hunt quests is still our intention, we did observe unintended results when players repeated this activity in combination with a specific multiplayer bug.
With this rewards adjustment we want to eliminate this exploit behaviour while still making that quest line worth the effort.
Players who took advantage of this exploit will not be punished.
Adjusted loot on specific enemies: Tooka Beast, Splittooth, Sandshifter and Coldclaw
These particular beasts are being too generous considering the required effort to take them down.
Chem Plant - Expedition time for Gold/Silver/Bronze adjusted to 650 / 1150 / 1500 seconds (Previously: 957 / 1303 / 1650)
Boom Town - Expedition time for Gold/Silver/Bronze adjusted to 390 / 589 / 776 seconds(Previously: 422 / 607 / 792)
Some Expeditions seem to be too generous in their required completion times when compared to others.
We want to avoid such disproportionate discrepancies, as we believe making all expeditions viable leads to a more diverse and thus more interesting endgame.
Critical Rating Damage Multiplier on Tactical Assault Rifle reduced to 165% (Previously: 175%)
Critical Rating Damage Multiplier on Marksman Rifle reduced to 250%(Previously: 300%)
Marksman Rifles with a 300% crit rate showed up to be dominating multiple crit builds. By making the modifier the same as for a Bolt Action Rifle, we want to offer players more options when choosing what gun to use.
Duration of the Massacre AP buff has been reduced to 3 seconds
This is a temporary measure until we have fixed an issue with the stacking cap.
Bullet abilities (Twisted Rounds, Blighted Rounds and Volcanic Rounds):
While we fully intend to provide the overpowered feel for the best builds out there, we are also committed to making sure that there is build diversity in the top performing builds.
Late in the demo lifecycle, we discovered a bug with regards to the bullet augmentation abilities damage calculation formula. While we fixed this bug, some unforeseen imbalances were unfortunately introduced, which has led to bullet based abilities vastly outperforming other skills. The main problem with this is that such bullet based builds require much less investment to make them "top tier", compared to other builds.
To achieve this, we have made a couple of adjustments for the Trickster and Technomancer classes, who benefit the most from their bullet abilities power. It's uncertain as to whether these changes will prove too strong or too weak, so please do consider this to be only the first step, as we will continue monitoring and adjusting them in the future. Just like you, we hope to see more inspiring and crazy combinations of Abilities and Mods!
The base cooldown of the Twisted Rounds skill has increased to 25 seconds(Previously: 16 seconds)
While infinite uptime on bullets abilities was always our intention, it is intended to be high risk, high reward playstyle. However, given the current power of this skill, there is very little risk involved in this ability given its short cooldown, even when players fail to trigger ammo replenishment. Increasing its cooldown in combination with adjusting its damage should increase that risk and make this playstyle a more thrilling experience.
The Trickster's Hero Tree nodes: - Disruptive Firepower, Scion of Power, Outrider Executioner - have had their power reduced to 35%(Previously: 50%)
The combination of Twisted Rounds' increase of firepower on top of the multipliers provided by the class tree lead to huge increases of damage dealt by this ability. While we do understand the satisfaction this kind of power can bring, it is currently vastly outperforming our wildest expectations and we therefore need to bring it down a notch.
"Vulnerable" status - Power reduced to 15% (Previously: 25%)
The Technomancer Hero Tree node -"Damage Against Poison" - has had its power reduced to 15%(Previously: 30%)
While the Technomancer's Blighted Rounds do not provide any damage increase on their own other than ignoring armor as Anomaly Damage, they do provide an excellent tool for spreading the Vulnerable and Toxic statuses. which in turn can scale out of proportion.
The Pyromancer Hero Tree Node - "Trial of Ashes" - has been reduced 15% extra damage (Previously: 30%)
First patch coming out next week -
Patch News:
First Patch:
This is our first big patch for the main Outriders game.
The deployment of this patch is currently scheduled for next week, as we are using this week to identify as many severe issues as possible, fix them and then thoroughly test the changes across all platforms. Submission through platform holders adds on a little bit of time as well, which is why we are not able to deploy this patch any earlier.
We are doing our utmost to deliver you a stable patch as soon as possible and we are continually looking at ways to bring the patch release day earlier. This is why we are not yet committing to a patch release date, as we may still be able to release the patch before such a date.
We will let you know as soon as we have a confirmed release schedule, so please do keep an eye on our social channels and in particular our Twitter.
First Patch - Patch Notes:
PC & Console Platforms:
Once platforms have been updated to the same patch version, cross-play across platforms will become viable again
Overall stability improvements for the matchmaking service
Crash Fixes
Will Fix a multiplayer crash that could result in client players having their inventory wiped
Will fix the crash when completing the "A Bad Day" side quest.
Will fix the crash that occurs in No Man's Land when your language is set to Spanish (Yes, we know. Video game code is a magical thing).
Will fix crash on launch issues
Will include many more "random" crash fixes
We are confident that these fixes will address the majority of crashes reported, as there are only a handful of root causes but the crashes they generate appear in a number of places.
Will fix the HUD disappearing in certain cases
Will fix bugs that interfere with players re-spawning in multiplayer Expeditions
Will fix bugs with players getting stuck on geometry (including when using Gravity Leap) or falling out of the world.
Will change the default matchmaking setting from "Open" to "Closed".
You will still be able to manually change this setting to "Open" through your game settings
This change will prevent players from joining games where the host didn't intend to play in multiplayer. It will also cut down on AFK lobbies
This will also help improve matchmaking times, as the queues will be less likely to be overwhelmed by the sheer volume of constant matchmaking requests generated by "open" games.
Many other minor fixes and improvements
PC Specific:
Will Fix a performance issue where GPU is not being fully utilized. This should help with stuttering and DX11/12 issues
Additional fixes and changes are being worked upon to be delivered via our later follow up patch.
Outriders Community Appreciation Package if you played before 11th April -
Your highest leveled character will receive:
A level appropriate Legendary Weapon
A level appropriate amount of Titanium
The emote "Frustration", which is otherwise unobtainable at this moment in time. The irony here was not intentional but is fitting.
https://www.reddit.com/r/outriders/comm ... s_updates/
.
