One of the main problems with refcounting is that unless your compiler/JIT are able to safely, aggressively optimize out reference increment/decrements, you can spend a ton of CPU time pointlessly bumping a counter up and down every time you enter a new function/method. This has been a problem for ObjC and Swift applications in the past AFAIK, though both of those compilers do a great job of optimizing that stuff out where possible.

There are some other things that would probably be improvements coming along with refcounting though - you might be able to get rid of GC write barriers.