Over the past year, Toptal data scientist and natural language processing engineer (NLP) Daniel Pérez Rubio has been intensely focused on developing advanced language models like BERT and GPT—the same language model family behind omnipresent generative AI technologies like OpenAI’s ChatGPT. What follows is a summary of a recent ask-me-anything-style Slack forum in which Rubio fielded questions about AI and NLP topics from other Toptal engineers around the world.

This comprehensive Q&A will answer the question “What does an NLP engineer do?” and satisfy your curiosity on subjects such as essential NLP foundations, recommended technologies, advanced language models, product and business concerns, and the future of NLP. NLP professionals of varying backgrounds can gain tangible insights from the topics discussed.

Editor’s note: Some questions and answers have been edited for clarity and brevity.

New to the Field: NLP Basics

What steps should a developer follow to move from working on standard applications to starting professional machine learning (ML) work?
—L.P., Córdoba, Argentina

Theory is much more important than practice in data science. Still, you’ll also have to get familiar with a new tool set, so I’d recommend starting with some online courses and trying to put your learnings into practice as much as possible. When it comes to programming languages, my recommendation is to go with Python. It’s similar to other high-level programming languages, offers a supportive community, and has well-documented libraries (another learning opportunity).

How familiar are you with linguistics as a formal discipline, and is this background helpful for NLP? What about information theory (e.g., entropy, signal processing, cryptanalysis)?
—V.D., Georgia, United States

As I am a graduate in telecommunications, information theory is the foundation that I use to structure my analytical approaches. Data science and information theory are considerably connected, and my background in information theory has helped shape me into the professional I am today. On the other hand, I have not had any kind of academic preparation in linguistics. However, I have always liked language and communication in general. I’ve learned about these topics through online courses and practical applications, allowing me to work alongside linguists in building professional NLP solutions.

Can you explain what BERT and GPT models are, including real-life examples?
—G.S.

Without going into too much detail, as there’s a lot of great literature on this topic, BERT and GPT are types of language models. They’re trained on plain text with tasks like text infilling, and are thus prepared for conversational use cases. As you have probably heard, language models like these perform so well that they can excel at many side use cases, like solving mathematical tests.