I think things would most likely be fine, but if you really want to test yourself, you should go to a gym.
How tough do you have to be to wing some wild shots with you eyes closed? That is what most of these "backyard boxing" matches devolve into.
Go to the gym and learn how to fight like a man, then you can be proud.
As for brain dammage, I wouldn't worry about that too much. Most fighters who suffer brain dammage have been fighting for 10+ years, and have taken hundreds of punches to the head in single fights, on multiple occasions. A couple school age scraps -- in the ring -- should be fine.
If you do feel the need for the "backyard boxing", make sure you fight on a grassy area that is clear of all rocks. NEVER fight on concrete -- if someone is knocked out they can easily fall and crack their head open.
Don't learn how to box so that you can street fight. A street fight is not a boxing match, it's a brutish sort of violence unrestrained by sportsmanship and fairness. Boxing may prepare you tactically for a street fight (you'll know how to throw a punch), but will leave you unprepared for the strategic elements of a street fight (you're actually going to be fighting 10 of these guys).
main point: Join a boxing gym.