As odd as it seams, sometimes, preventing RecyclerView from doing what it was created to do is a better option if you don't want some views messed up