Definitely at the checkpoints. As Bro Yonglip said, this will be "mother of all jams". If you can, avoid this weekend otherwise you can safely claim to have been through the worst! Most of the SPR (who has relatives in Peninsula Malaysia) will be heading back regardless of races because of long weekend and schools are off too. If you are in JB, places will be generally more crowded including eateries.
I thought the jam is only at teh causeway/2nd link /immigration can you confirm this?
Or is it even within skudai/Johor/Nusajaya area?