So let me break down the decision process at several places I've worked:
For the purposes of this post, I'm mainly referring to third-party tools/systems, open source or otherwise. Think of something like Jenkins except actually good.
In my experience, it is usually better to make a serious effort fixing the existing system (or your usage of the existing system) than it is to implement a new one. This is driven by several observations:
The old system was probably not set up correctly.
One reason you may be thinking of replacing something is that the old system is a dumpster fire set up by you or your predecessors. It crashes all the time, when it is working it takes two minutes to respond to any request, and if it were to really fall over, no one knows how to make a new one from scratch. Yet the "worst thing ever" might not be that bad if someone took the time to properly configure it (JVM tuning, getting the proper storage, allocating system resources, enabling the flag that says "don't be slow as shit," etc.). It's amazing how not using the default sqlite3/whatever storage backend can suddenly make things better. It's also likely this thing has no monitoring, tests, HA, and/or recovery plans. This just adds to the impression that the system itself is bad and not just the way it was configured.
If you think needing to know all this stuff makes this tool awful, I have a spoiler alert for you:
The new system will not be set up correctly, either.
Unless you've taken all the steps necessary properly configure the old system before you've arrived at "we'll use something else," you probably are just going to make the same mistakes setting up the new system. There's usually a gulf between "we want this new thing" and "we want this new thing and have done all the research into what goes into setting it up and keeping it running." Moving goal posts is something done by both management and engineering. It is likely the case that the new system being introduced is a means to an end, and not the primary concern of the team. "We are adding this new X to unblock Y." We often forget that this means X will also need to be maintained.
Here's a good approach: The less you care about something that is your responsibility, the more robust it needs to be.
But wait! We're going to replace this with a hosted system (SaaS).
There are users of the existing system.
This should probably be extended to say "there are users of the existing system, and some of them are using it in ways you don't realize and you have no idea how to support them."
These people are your customers (developers/managers/etc). They rely on it to get some aspect of their job done. There are workflows (official and unofficial) that have been implemented around the thing you have today. Unless the thing you are replacing the old system with has the same interface, these customers are going to have to change the way they work.
Change isn't always a bad thing, but we often overlook how much a "minor" change can impact productivity beyond just the adjustment period of learning a new tool. The closer the system is to a choke point in developer productivity, the larger of an impact even small changes will have.
There is often a disconnect between how much you care about something and how much you think someone else cares about it. What may be your passion is just a means to an end to other people, and it's important to find balance here. Doing the job we are hired to do sometimes means accepting that we can't just use the tool we're used to or the one we wished we had.
A classic example is wanting to swap out a CI system because the existing one is "slow," or "can't do X in a very specific way," or "is ugly." Adjusting to a new interface is something we should all be able to do, but it often takes time. Just because the new thing does X better doesn't mean nothing else gets worse. Having worked with it several times now, I think Jenkins is a pain to work with (inconsistent API, really easy to set up incorrectly, performance degrades when retaining job history for many projects, etc.), but if I were to start at a new company that was using Jenkins, there would have to be a significantly compelling reason to propose swapping it out for something else. Assuming we're in the boat of "the CI system is slow," this implies jobs aren't configured in code (so replicating all the job definitions is already going to be a lot of work), they're running a bunch of plugins (whose functionality now has to be re-implemented or equivalents found), and there is probably significant dependence on the Jenkins UI itself.
Working with the existing system instead of completely replacing it means it is easier to support existing workflows while promoting future ones. This leads to:
Transition periods are always longer than you expect.
There will always be a transition period between the "old" system and the "new" system. This transition period will be longer than you anticipated. The cost of implementing a new system to replace the old one needs to be weighed against the cost of maintaining both systems for an extended period of time. It might be nice to think that once the new system is working, you can make everyone switch overnight, but even if this was the case, there's still the period of time in which you're verifying the new system isn't worse than the old one. The exception would be a transition to a SaaS thing, but even that only cuts down on maintenance of the new system. It's likely the old system would still be up for some time.
So...
If you are going to swap something old for something "new," make sure you know exactly the problem you're trying to solve, exactly why the old thing is bad (and can't just be fixed), how to set up the new thing correctly, all the use cases you need to support, and be prepared to support both things for a while. And know it's usually much more work.