Genre fiction is many things. There are mysteries, suspense, romance, fantasy. Each of these categories can have a subgenre. These genres also can combine. Take a suspense story and add a heroine to the hero's quest to fine the villain and the story changes slightly.
Mainstream fiction is looking at best-seller status. There can be elements of genres in the story. This type of story isn't comfortable being just a genre book.
Experimental fiction breaks the rules in some way. I've attempted this kind of story and failed.The writers of this type of fiction are interested in form.