Well, actually i didnt mind the drums. Its the combination of both. The drums are weird and experimental but i dont think their bad. Its just that with that pattern you really need to find something thats suits it which is gonna be harder then a 1,4,5 kick, kick, snare pattern.
I found them to be rather "score" like and think you can really do something with it
One thing that might work as well is lower the sidechain on that fifth type synth halftime to give it a halftime contrast, so youll get a fast pace drum pattern in conjunction with the slower paced synth sidechain wich will give it some anticipation.
Maybe a slower underlying dark melody will work well?
The drums remind me of old "aes dana's" tracks, maybe worth checking out. I think the album "leylines" has some similar stuff going on. Maybe worth checking out