It is possible you have a setting turned on which adds addresses automatically as you reply to group/bulk emails? If so, all the recipients may be getting added by the system.
Send an email to an email address which is definitely NOT in your contacts and see if that address gets added automatically.
If that confirms it, look in your settings for an 'Add recipients to contacts' settings, or similar.