Cognition of space and time affect each other; a line with longer length appears to be longer in exposure duration (space on time), and a line with longer exposure duration appears to be longer in length (time on space). This cognitive interaction is known to be asymmetric; the effect of space on time is larger than that of time on space. We conjectured that this asymmetry is not intrinsic but may depend on the saliency of relevant signals. Participants were asked to judge the visual exposure duration of lines that varied in length or the lengths of the lines with different exposure times. The ranges of task-relevant and -irrelevant stimulus values were the same in the spatial and temporal tasks. Task difficulty was also evaluated by subjective rating. We found that duration affected the judgment of length more than vice versa, when the spatial task was significantly more difficult than the temporal task. Together with our previous results that showed the opposite effect, our conjecture is supported that the saliency of stimuli should affect the balance of interactions.