Shabbas 30b – answering a mamzerus insult

We learned that both Rebbi and Rabbi Chiya were a proof to the statement that when it comes to non Torah statements, אלתען כסיל כאולתו, i.e. – don’t answer and debate him. The Ben Ish Chai in his sefer Ben Yehoyada says he found a worn piece of paper with a question asked by his grandfather, from the gemara in bava metzia that we learned – where Rabbi Shimon Ben Elazar was suspected as not having fathered his own children, due to his girth. But over there, he did answer – why?

Ben Yehoyada answers that there it was not a real accusation, it was just השערת שכל and therefore, since Rabbi Shimon Ben Elazar had a concrete answer to her suggestion, it was appropriate to disprove her. But in this case, the guy was making a lying accusation  and there is no way to really prove him, therefore it was best to not answer and get him punished min hashamayim.