Usually, books bought for the Nook are sold by Barnes & Noble and are sold with DRM (Digital Rights Management) files attached. So, they will only work with the Nook app.
The odds are that those new books have that DRM on them, that's why you can't open them with the iBooks app. I should imagine that the ones that ARE working in iBooks don't have DRM.
It's a publishing thing, in that publishers load up e-books with DRM so that the e-book can only be read on the bookseller's proprietary e-reader or app. They all do it - Amazon, Barnes & Noble, Borders, Kobo, etc. You just have to use their app and it'll be fine. A bit cumbersome, yeah, remembering which app has which book, but, that's the way it is with DRM.
Hope this helped in some way, though it wasn't very encouraging.
Marilyn