Is

    fossil setting mtime-changes

turned on?  If not, Fossil has to compute a SHA1 hash for every file in the
project in order to see which ones have changed.

If that isn't the problem, can you run fossil in a profiler to give me a better
idea of what the problem might be?