I'm really over the hype around BERT, it's just a static context window and some dense layers, not some architecture.