- One of the risks associated with foundation models is their ever-increasing scale.
- Pre-training on a large general-purpose dataset is no guarantee that the network will be able to perform a new task on proprietary data.
- Dataset alignment and recency of training data can matter immensely depending on the use case.