First, which version are you using? We are doing a lot of work on a new design. 1.0 version is the start for that, much more is coming.
Currently I'm using 0.16.8, but saw that there's version 1.0 already, and it'd be fine to aim just that.
Now I'm not sure to understand what you want to do exactly. If you want to write acompletly new UI (in curses or QT or whatever) then that should be possible. "Just" replace all files in src by the ui design you want. common folder is independant of GUI.
Explain a bit more what you want to change in the UI, we can be more precise.
I thought of using D-Bus to reuse Gajim functionality from Emacs, implementing the UI itself inside of Emacs (so that all the Emacs text editing and navigation commands and packages would be available). Seems like it would be rather easy, at least to get basic functionality, if there were a few additional D-Bus events available; that wouldn't involve rewriting/replacing UI bits in Gajim itself. Somewhat similar in approach to https://github.com/the-kenny/weechat.el.
But as forenjunkie suggested, perhaps it indeed would be easier to pick some library, make a basic client providing D-Bus interface and no UI, and then proceed with that external UI.