I’ve recently refined using taskwarrior a lot more, and had to finally do something about annotations. I sometimes need to leave a bit of info, or a relevant URL, on a task, but that bumps up the urgency:
The lowest coefficient, that of annotations, has a value of 1.0. This means that a task with more annotations is more urgent than a task with fewer. The value of 1.0 means this is a weak contributor to urgency, but nevertheless means that tasks with lots of annotations are an indicator of some kind of focus of activity. You may not agree with this assessment however, and so you might set this coefficient to 0, which means you want the term to have no effect on urgency.
— Taskwarrior - Urgency - Taskwarrior
So I added urgency.annotations.coefficient=0.0 to my .taskrc and that is no longer an issue. 