On ours, the front roller needs a brutal amount of force to release. Could it be that it's not engaging properly because you're not turning it hard enough? Otherwise, since it's obviously not the roller (wear material?) but the building, I would try it as a warranty claim. You have probably swapped the two rollers before.

We need a little force to engage the castors. However, it takes a lot to release them. I can hardly release it with my hands. I always use combination pliers to help me. It's really easy that way. But don't forget to clean the lower part regularly by hand. Sometimes the rollers get stuck because we only used the automatic cleaning function at the beginning. But that's not enough.

We have the same problem.