Actually, if you read a kind of retraction at Slate, you'd know that Scalise did not speak to Duke's group at all,  He spoke to a group of law makers at the same location of the planned Duke group but at a different time.  It had nothing to do with racism at all on his part. 

That he didn't remember the day and speech in question, led to a lot of confusion and made him look guilty of the claim.  It is a false claim but there is little likelihood that progressive sources will drop the bone on this one.  They'll keep ragging on about it for as long as it has shelf life.