Remove unwanted filters cluttering filter pane

Right now I often have around 10 fields incorporated in a visual. I like to include alot of relevant data points as tooltips. Because every used field jumps to the filter pane, the filter pane gets very cluttered with filters I do not care about. For instance, when I create a map using latitude and longitude, there are automatically filters for these fields. I will never want to filter based on these fields. I find myself having to scroll way down to the bottom of the filter page to drag/drop desired fields into the filter pane.

Additionally, I usually want to use the basic filter feature, but often the default filter that appears due to the addition of a tooltip will only allow advanced filtering. This means I also have to add certain fields as page filters. This then gets very confusing and cluttered as I have 2 near identical filters on the same pane.