I'm not sure if this is the correct forum--Moderators, please move if appropriate.

I have a vexing problem with someone sending spam with fake "from" email addresses supposedly at my domain (i.e. "ehdacywjkg@bigblueball.com"). We have a mailserver, but we do not provide an open SMTP service, and checking the logs it's clear that they aren't actually being sent from our server. But my catch-all account is getting the hundreds of bounces that their spam is producing.

I'm concerned that this action might lead to my domain being blacklisted and the domain reputation tarnished by this spam.

Is this a real concern? Is there anything I can do about it? How do I know if my domain does end up blacklisted by an ISP? Is there any way to fight back against this insidious abuse?