Metal and dielectric nanostructure-based optical circulators and isolators to direct light through optical circuits