If you happen to have that opinion of the series, then I urge you to do two things:
1) Read the series yourself if you haven't already. Please don't judge based on hearsay or the opinion of others.
2) If you've already read the books and still find yourself on the fence, please read A Parent's Guide to Harry Potter. It just might change your mind about the series.
Gina Burkart, author of the book, does an excellent job of laying out the themes found in the books, and subsequently showing that these are the heart of the books, not the magical aspects.
The first section of the book discusses what children can learn from Harry Potter (i.e., morals). She also offers "Questions to bridge the gap" which parents can use to discuss sometimes difficult issues with their children - situations they may face in school with other children, or even handling bad dreams (as Harry does throughout the series).
The second section of the book deals with how to talk about Harry Potter with your children. How, as parents, do you go about discussing the bullying that occurs, fears, or making tough choices? Burkart lays it all out for us, and does so exceptionally well. She points time and again to the fact that love is the all-encompassing aspect of the book. Love is what brought Harry into the world and allowed him to survive, beating the odds.
Burkart also discusses how to talk to your kids about the difference between fantasy and reality, which is important with many popular series that they might one day want to read (The Chronicles of Narnia and The Lord of the Rings come to my mind immediately).
I'm obviously an avid supporter of Harry Potter. I sincerely believe the series offers so much for kids . . . and adults! Not only do they profess love, but also hope. And couldn't we all use a little more hope in our daily lives? I encourage you to check out A Parent's Guide to Harry Potter if you're experiencing any misgivings about letting your kids read the books. I don't think you'll be disappointed.