I'd go with Watsonsimler's suggestion because it reduces the chance of the user accidentally clicking the delete button in a row they don't want to delete.
You can't also have a checkBox cell in every row and a single delete button on your form (not in the data grid view). This will let the user select multiple rows to delete at once. If you use the checkBox option you can add buttons to do other functions such as exporting the selected rows to an Excel document or something like that.
Sorry I went off on one then but it's another option you could think about which could reduce the amount of work you'll need to do if something like that became a requirement in the future.